Merge tag 'v3.5.0' into main
2022-08-31 v3.5.0
This release is ABI compatible with the last one, including speedup and memory
optimizations, and new APIs and features.
- New Features
* Support for frame parallel encode for larger number of threads. --fp-mt
flag is available for all build configurations.
* New codec control AV1E_GET_NUM_OPERATING_POINTS
- Speedup and Memory Optimizations
* Speed-up multithreaded encoding for good quality mode for larger number of
threads through frame parallel encoding:
o 30-34% encode time reduction for 1080p, 16 threads, 1x1 tile
configuration (tile_rows x tile_columns)
o 18-28% encode time reduction for 1080p, 16 threads, 2x4 tile
configuration
o 18-20% encode time reduction for 2160p, 32 threads, 2x4 tile
configuration
* 16-20% speed-up for speed=6 to 8 in still-picture encoding mode
* 5-6% heap memory reduction for speed=6 to 10 in real-time encoding mode
* Improvements to the speed for speed=7, 8 in real-time encoding mode
* Improvements to the speed for speed=9, 10 in real-time screen encoding
mode
* Optimizations to improve multi-thread efficiency in real-time encoding
mode
* 10-15% speed up for SVC with temporal layers
* SIMD optimizations:
o Improve av1_quantize_fp_32x32_neon() 1.05x to 1.24x faster
o Add aom_highbd_quantize_b{,_32x32,_64x64}_adaptive_neon() 3.15x to 5.6x
faster than "C"
o Improve av1_quantize_fp_64x64_neon() 1.17x to 1.66x faster
o Add aom_quantize_b_avx2() 1.4x to 1.7x faster than aom_quantize_b_avx()
o Add aom_quantize_b_32x32_avx2() 1.4x to 2.3x faster than
aom_quantize_b_32x32_avx()
o Add aom_quantize_b_64x64_avx2() 2.0x to 2.4x faster than
aom_quantize_b_64x64_ssse3()
o Add aom_highbd_quantize_b_32x32_avx2() 9.0x to 10.5x faster than
aom_highbd_quantize_b_32x32_c()
o Add aom_highbd_quantize_b_64x64_avx2() 7.3x to 9.7x faster than
aom_highbd_quantize_b_64x64_c()
o Improve aom_highbd_quantize_b_avx2() 1.07x to 1.20x faster
o Improve av1_quantize_fp_avx2() 1.13x to 1.49x faster
o Improve av1_quantize_fp_32x32_avx2() 1.07x to 1.54x faster
o Improve av1_quantize_fp_64x64_avx2() 1.03x to 1.25x faster
o Improve av1_quantize_lp_avx2() 1.07x to 1.16x faster
- Bug fixes including but not limited to
* aomedia:3206 Assert that skip_width > 0 for deconvolve function
* aomedia:3278 row_mt enc: Delay top-right sync when intraBC is enabled
* aomedia:3282 blend_a64_*_neon: fix bus error in armv7
* aomedia:3283 FRAME_PARALLEL: Propagate border size to all cpis
* aomedia:3283 RESIZE_MODE: Fix incorrect strides being used for motion
search
* aomedia:3286 rtc-svc: Fix to dynamic_enable spatial layers
* aomedia:3289 rtc-screen: Fix to skipping inter-mode test in nonrd
* aomedia:3289 rtc-screen: Fix for skip newmv on flat blocks
* aomedia:3299 Fix build failure with CONFIG_TUNE_VMAF=1
* aomedia:3296 Fix the conflict --enable-tx-size-search=0 with nonrd mode
--enable-tx-size-search will be ignored in non-rd pick mode
* aomedia:3304 Fix off-by-one error of max w/h in validate_config
* aomedia:3306 Do not use pthread_setname_np on GNU/Hurd
* aomedia:3325 row-multithreading produces invalid bitstream in some cases
* chromium:1346938, chromium:1338114
* compiler_flags.cmake: fix flag detection w/cmake 3.17-3.18.2
* tools/*.py: update to python3
* aom_configure.cmake: detect PIE and set CONFIG_PIC
* test/simd_cmp_impl: use explicit types w/CompareSimd*
* rtc: Fix to disable segm for aq-mode=3
* rtc: Fix to color_sensitivity in variance partition
* rtc-screen: Fix bsize in model rd computation for intra chroma
* Fixes to ensure the correct behavior of the encoder algorithms (like
segmentation, computation of statistics, etc.)
Bug: aomedia:3313
Change-Id: I8c9bc4c709f3bf0157ec29c5af52f397ac33ec38
diff --git a/.clang-format b/.clang-format
index a378820..a8bc496 100644
--- a/.clang-format
+++ b/.clang-format
@@ -1,148 +1,9 @@
---
Language: Cpp
-# BasedOnStyle: Google
-# Generated with clang-format 7.0.1
-AccessModifierOffset: -1
-AlignAfterOpenBracket: Align
-AlignConsecutiveAssignments: false
-AlignConsecutiveDeclarations: false
-AlignEscapedNewlines: Left
-AlignOperands: true
-AlignTrailingComments: true
-AllowAllParametersOfDeclarationOnNextLine: true
-AllowShortBlocksOnASingleLine: false
+BasedOnStyle: Google
AllowShortCaseLabelsOnASingleLine: true
-AllowShortIfStatementsOnASingleLine: true
-AllowShortLoopsOnASingleLine: true
-AlwaysBreakAfterDefinitionReturnType: None
-AlwaysBreakAfterReturnType: None
-AlwaysBreakBeforeMultilineStrings: true
-AlwaysBreakTemplateDeclarations: true
-BinPackArguments: true
-BinPackParameters: true
-BraceWrapping:
- AfterClass: false
- AfterControlStatement: false
- AfterEnum: false
- AfterFunction: false
- AfterNamespace: false
- AfterObjCDeclaration: false
- AfterStruct: false
- AfterUnion: false
- AfterExternBlock: false
- BeforeCatch: false
- BeforeElse: false
- IndentBraces: false
- SplitEmptyFunction: true
- SplitEmptyRecord: true
- SplitEmptyNamespace: true
-BreakBeforeBinaryOperators: None
-BreakBeforeBraces: Attach
-BreakBeforeInheritanceComma: false
-BreakInheritanceList: BeforeColon
-BreakBeforeTernaryOperators: true
-BreakConstructorInitializersBeforeComma: false
-BreakConstructorInitializers: BeforeColon
-BreakAfterJavaFieldAnnotations: false
-BreakStringLiterals: true
-ColumnLimit: 80
-CommentPragmas: '^ IWYU pragma:'
-CompactNamespaces: false
ConstructorInitializerAllOnOneLineOrOnePerLine: false
-ConstructorInitializerIndentWidth: 4
-ContinuationIndentWidth: 4
Cpp11BracedListStyle: false
DerivePointerAlignment: false
-DisableFormat: false
-ExperimentalAutoDetectBinPacking: false
-FixNamespaceComments: true
-ForEachMacros:
- - foreach
- - Q_FOREACH
- - BOOST_FOREACH
-IncludeBlocks: Preserve
-IncludeCategories:
- - Regex: '^<ext/.*\.h>'
- Priority: 2
- - Regex: '^<.*\.h>'
- Priority: 1
- - Regex: '^<.*'
- Priority: 2
- - Regex: '.*'
- Priority: 3
-IncludeIsMainRegex: '([-_](test|unittest))?$'
-IndentCaseLabels: true
-IndentPPDirectives: None
-IndentWidth: 2
-IndentWrappedFunctionNames: false
-JavaScriptQuotes: Leave
-JavaScriptWrapImports: true
-KeepEmptyLinesAtTheStartOfBlocks: false
-MacroBlockBegin: ''
-MacroBlockEnd: ''
-MaxEmptyLinesToKeep: 1
-NamespaceIndentation: None
-ObjCBinPackProtocolList: Never
-ObjCBlockIndentWidth: 2
-ObjCSpaceAfterProperty: false
-ObjCSpaceBeforeProtocolList: false
-PenaltyBreakAssignment: 2
-PenaltyBreakBeforeFirstCallParameter: 1
-PenaltyBreakComment: 300
-PenaltyBreakFirstLessLess: 120
-PenaltyBreakString: 1000
-PenaltyBreakTemplateDeclaration: 10
-PenaltyExcessCharacter: 1000000
-PenaltyReturnTypeOnItsOwnLine: 200
PointerAlignment: Right
-RawStringFormats:
- - Language: Cpp
- Delimiters:
- - cc
- - CC
- - cpp
- - Cpp
- - CPP
- - 'c++'
- - 'C++'
- CanonicalDelimiter: ''
- BasedOnStyle: google
- - Language: TextProto
- Delimiters:
- - pb
- - PB
- - proto
- - PROTO
- EnclosingFunctions:
- - EqualsProto
- - EquivToProto
- - PARSE_PARTIAL_TEXT_PROTO
- - PARSE_TEST_PROTO
- - PARSE_TEXT_PROTO
- - ParseTextOrDie
- - ParseTextProtoOrDie
- CanonicalDelimiter: ''
- BasedOnStyle: google
-ReflowComments: true
SortIncludes: false
-SortUsingDeclarations: true
-SpaceAfterCStyleCast: false
-SpaceAfterTemplateKeyword: true
-SpaceBeforeAssignmentOperators: true
-SpaceBeforeCpp11BracedList: false
-SpaceBeforeCtorInitializerColon: true
-SpaceBeforeInheritanceColon: true
-SpaceBeforeParens: ControlStatements
-SpaceBeforeRangeBasedForLoopColon: true
-SpaceInEmptyParentheses: false
-SpacesBeforeTrailingComments: 2
-SpacesInAngles: false
-SpacesInContainerLiterals: false
-SpacesInCStyleCastParentheses: false
-SpacesInParentheses: false
-SpacesInSquareBrackets: false
-Standard: Auto
-TabWidth: 8
-UseTab: Never
-...
-
diff --git a/.cmake-format.py b/.cmake-format.py
index 7b0e4f0..c79a6ad 100644
--- a/.cmake-format.py
+++ b/.cmake-format.py
@@ -64,7 +64,7 @@
# If comment markup is enabled, don't reflow the first comment block in
# eachlistfile. Use this to preserve formatting of your
# copyright/licensestatements.
-first_comment_is_literal = False
+first_comment_is_literal = True
# If comment markup is enabled, don't reflow any comment block which matchesthis
# (regex) pattern. Default is `None` (disabled).
diff --git a/README.md b/README.md
index 0146003..d70c707 100644
--- a/README.md
+++ b/README.md
@@ -19,16 +19,16 @@
- [Build with VMAF support](#build-with-vmaf)
2. [Testing the library](#testing-the-av1-codec)
- [Basics](#testing-basics)
- - [Unit tests](#1_unit-tests)
- - [Example tests](#2_example-tests)
- - [Encoder tests](#3_encoder-tests)
+ - [Unit tests](#unit-tests)
+ - [Example tests](#example-tests)
+ - [Encoder tests](#encoder-tests)
- [IDE hosted tests](#ide-hosted-tests)
- [Downloading test data](#downloading-the-test-data)
- [Adding a new test data file](#adding-a-new-test-data-file)
- [Additional test data](#additional-test-data)
- [Sharded testing](#sharded-testing)
- - [Running tests directly](#1_running-test_libaom-directly)
- - [Running tests via CMake](#2_running-the-tests-via-the-cmake-build)
+ - [Running tests directly](#running-test_libaom-directly)
+ - [Running tests via CMake](#running-the-tests-via-the-cmake-build)
3. [Coding style](#coding-style)
4. [Submitting patches](#submitting-patches)
- [Login cookie](#login-cookie)
@@ -341,7 +341,7 @@
the presence of the AV1 source code and a working build of the AV1 library and
applications.
-#### 1. Unit tests: {#1_unit-tests}
+#### 1. Unit tests: {#unit-tests}
The unit tests can be run at build time:
@@ -355,7 +355,7 @@
$ make runtests
~~~
-#### 2. Example tests: {#2_example-tests}
+#### 2. Example tests: {#example-tests}
The example tests require a bash shell and can be run in the following manner:
@@ -370,7 +370,7 @@
$ path/to/aom/test/examples.sh --bin-path examples
~~~
-#### 3. Encoder tests: {#3_encoder-tests}
+#### 3. Encoder tests: {#encoder-tests}
When making a change to the encoder run encoder tests to confirm that your
change has a positive or negligible impact on encode quality. When running these
@@ -487,7 +487,7 @@
The AV1 codec library unit tests are built upon gtest which supports sharding of
test jobs. Sharded test runs can be achieved in a couple of ways.
-#### 1. Running test\_libaom directly: {#1_running-test_libaom-directly}
+#### 1. Running test\_libaom directly: {#running-test_libaom-directly}
~~~
# Set the environment variable GTEST_TOTAL_SHARDS to control the number of
@@ -501,7 +501,7 @@
To create a test shard for each CPU core available on the current system set
`GTEST_TOTAL_SHARDS` to the number of CPU cores on your system minus one.
-#### 2. Running the tests via the CMake build: {#2_running-the-tests-via-the-cmake-build}
+#### 2. Running the tests via the CMake build: {#running-the-tests-via-the-cmake-build}
~~~
# For IDE based builds, ENABLE_IDE_TEST_HOSTING must be enabled. See
diff --git a/aom/aomcx.h b/aom/aomcx.h
index 8f129369..37516fa 100644
--- a/aom/aomcx.h
+++ b/aom/aomcx.h
@@ -191,8 +191,8 @@
/* NOTE: enum 10 unused */
- /*!\brief Codec control function to set encoder scaling mode,
- * aom_scaling_mode_t* parameter.
+ /*!\brief Codec control function to set encoder scaling mode for the next
+ * frame to be coded, aom_scaling_mode_t* parameter.
*/
AOME_SET_SCALEMODE = 11,
diff --git a/aom_dsp/aom_dsp.cmake b/aom_dsp/aom_dsp.cmake
index 7e2f570..29a9d3f 100644
--- a/aom_dsp/aom_dsp.cmake
+++ b/aom_dsp/aom_dsp.cmake
@@ -250,7 +250,9 @@
"${AOM_ROOT}/aom_dsp/x86/jnt_variance_ssse3.c"
"${AOM_ROOT}/aom_dsp/x86/jnt_sad_ssse3.c")
- list(APPEND AOM_DSP_ENCODER_INTRIN_SSE4_1 "${AOM_ROOT}/aom_dsp/x86/sse_sse4.c"
+ list(APPEND AOM_DSP_ENCODER_INTRIN_SSE4_1
+ "${AOM_ROOT}/aom_dsp/x86/avg_intrin_sse4.c"
+ "${AOM_ROOT}/aom_dsp/x86/sse_sse4.c"
"${AOM_ROOT}/aom_dsp/x86/obmc_sad_sse4.c"
"${AOM_ROOT}/aom_dsp/x86/obmc_variance_sse4.c")
diff --git a/aom_dsp/aom_dsp_rtcd_defs.pl b/aom_dsp/aom_dsp_rtcd_defs.pl
index 4503c8e..d672d77 100755
--- a/aom_dsp/aom_dsp_rtcd_defs.pl
+++ b/aom_dsp/aom_dsp_rtcd_defs.pl
@@ -764,30 +764,30 @@
}
add_proto qw/uint64_t aom_sum_sse_2d_i16/, "const int16_t *src, int src_stride, int width, int height, int *sum";
- specialize qw/aom_sum_sse_2d_i16 sse2 avx2/;
+ specialize qw/aom_sum_sse_2d_i16 avx2 neon sse2/;
specialize qw/aom_sad128x128 avx2 neon sse2/;
- specialize qw/aom_sad128x64 avx2 sse2/;
- specialize qw/aom_sad64x128 avx2 sse2/;
+ specialize qw/aom_sad128x64 avx2 neon sse2/;
+ specialize qw/aom_sad64x128 avx2 neon sse2/;
specialize qw/aom_sad64x64 avx2 neon msa sse2/;
- specialize qw/aom_sad64x32 avx2 msa sse2/;
- specialize qw/aom_sad32x64 avx2 msa sse2/;
+ specialize qw/aom_sad64x32 avx2 neon msa sse2/;
+ specialize qw/aom_sad32x64 avx2 neon msa sse2/;
specialize qw/aom_sad32x32 avx2 neon msa sse2/;
- specialize qw/aom_sad32x16 avx2 msa sse2/;
- specialize qw/aom_sad16x32 msa sse2/;
+ specialize qw/aom_sad32x16 avx2 neon msa sse2/;
+ specialize qw/aom_sad16x32 neon msa sse2/;
specialize qw/aom_sad16x16 neon msa sse2/;
specialize qw/aom_sad16x8 neon msa sse2/;
specialize qw/aom_sad8x16 neon msa sse2/;
specialize qw/aom_sad8x8 neon msa sse2/;
- specialize qw/aom_sad8x4 msa sse2/;
- specialize qw/aom_sad4x8 msa sse2/;
+ specialize qw/aom_sad8x4 neon msa sse2/;
+ specialize qw/aom_sad4x8 neon msa sse2/;
specialize qw/aom_sad4x4 neon msa sse2/;
- specialize qw/aom_sad4x16 sse2/;
- specialize qw/aom_sad16x4 sse2/;
- specialize qw/aom_sad8x32 sse2/;
- specialize qw/aom_sad32x8 sse2/;
- specialize qw/aom_sad16x64 sse2/;
- specialize qw/aom_sad64x16 sse2/;
+ specialize qw/aom_sad4x16 neon sse2/;
+ specialize qw/aom_sad16x4 neon sse2/;
+ specialize qw/aom_sad8x32 neon sse2/;
+ specialize qw/aom_sad32x8 neon sse2/;
+ specialize qw/aom_sad16x64 neon sse2/;
+ specialize qw/aom_sad64x16 neon sse2/;
specialize qw/aom_sad_skip_128x128 avx2 sse2 neon/;
specialize qw/aom_sad_skip_128x64 avx2 sse2 neon/;
@@ -1002,34 +1002,31 @@
add_proto qw/void/, "aom_masked_sad${w}x${h}x4d", "const uint8_t *src, int src_stride, const uint8_t *ref[4], int ref_stride, const uint8_t *second_pred, const uint8_t *msk, int msk_stride, int invert_mask, unsigned sads[4]";
}
- specialize qw/aom_sad128x128x4d avx2 sse2/;
- specialize qw/aom_sad128x64x4d avx2 sse2/;
- specialize qw/aom_sad64x128x4d avx2 sse2/;
+ specialize qw/aom_sad128x128x4d avx2 neon sse2/;
+ specialize qw/aom_sad128x64x4d avx2 neon sse2/;
+ specialize qw/aom_sad64x128x4d avx2 neon sse2/;
specialize qw/aom_sad64x64x4d avx2 neon msa sse2/;
- specialize qw/aom_sad64x32x4d avx2 msa sse2/;
- specialize qw/aom_sad64x16x4d avx2 sse2/;
- specialize qw/aom_sad32x64x4d avx2 msa sse2/;
+ specialize qw/aom_sad64x32x4d avx2 neon msa sse2/;
+ specialize qw/aom_sad32x64x4d avx2 neon msa sse2/;
specialize qw/aom_sad32x32x4d avx2 neon msa sse2/;
- specialize qw/aom_sad32x16x4d avx2 msa sse2/;
- specialize qw/aom_sad32x8x4d avx2 sse2/;
- specialize qw/aom_sad16x64x4d sse2/;
- specialize qw/aom_sad16x32x4d msa sse2/;
- specialize qw/aom_sad16x16x4d neon msa sse2/;
- specialize qw/aom_sad16x8x4d msa sse2/;
+ specialize qw/aom_sad32x16x4d avx2 neon msa sse2/;
+ specialize qw/aom_sad16x32x4d neon msa sse2/;
+ specialize qw/aom_sad16x16x4d neon msa sse2/;
+ specialize qw/aom_sad16x8x4d neon msa sse2/;
- specialize qw/aom_sad8x16x4d msa sse2/;
- specialize qw/aom_sad8x8x4d msa sse2/;
- specialize qw/aom_sad8x4x4d msa sse2/;
- specialize qw/aom_sad4x16x4d msa sse2/;
- specialize qw/aom_sad4x8x4d msa sse2/;
- specialize qw/aom_sad4x4x4d msa sse2/;
+ specialize qw/aom_sad8x16x4d neon msa sse2/;
+ specialize qw/aom_sad8x8x4d neon msa sse2/;
+ specialize qw/aom_sad8x4x4d neon msa sse2/;
+ specialize qw/aom_sad4x32x4d neon sse2/;
+ specialize qw/aom_sad4x8x4d neon msa sse2/;
+ specialize qw/aom_sad4x4x4d neon msa sse2/;
- specialize qw/aom_sad4x32x4d sse2/;
- specialize qw/aom_sad4x16x4d sse2/;
- specialize qw/aom_sad16x4x4d sse2/;
- specialize qw/aom_sad8x32x4d sse2/;
- specialize qw/aom_sad32x8x4d sse2/;
- specialize qw/aom_sad64x16x4d sse2/;
+ specialize qw/aom_sad64x16x4d avx2 neon sse2/;
+ specialize qw/aom_sad32x8x4d avx2 neon sse2/;
+ specialize qw/aom_sad16x64x4d neon sse2/;
+ specialize qw/aom_sad16x4x4d neon sse2/;
+ specialize qw/aom_sad8x32x4d neon sse2/;
+ specialize qw/aom_sad4x16x4d neon msa sse2/;
specialize qw/aom_sad_skip_128x128x4d avx2 sse2 neon/;
specialize qw/aom_sad_skip_128x64x4d avx2 sse2 neon/;
@@ -1046,15 +1043,12 @@
specialize qw/aom_sad_skip_16x32x4d sse2 neon/;
specialize qw/aom_sad_skip_16x16x4d sse2 neon/;
specialize qw/aom_sad_skip_16x8x4d sse2 neon/;
+ specialize qw/aom_sad_skip_8x32x4d sse2 neon/;
specialize qw/aom_sad_skip_8x16x4d sse2 neon/;
specialize qw/aom_sad_skip_8x8x4d sse2 neon/;
- specialize qw/aom_sad_skip_4x16x4d sse2 neon/;
- specialize qw/aom_sad_skip_4x8x4d sse2 neon/;
specialize qw/aom_sad_skip_4x32x4d sse2 neon/;
specialize qw/aom_sad_skip_4x16x4d sse2 neon/;
- specialize qw/aom_sad_skip_8x32x4d sse2 neon/;
- specialize qw/aom_sad_skip_32x8x4d sse2 neon/;
- specialize qw/aom_sad_skip_64x16x4d sse2 neon/;
+ specialize qw/aom_sad_skip_4x8x4d sse2 neon/;
if (aom_config("CONFIG_REALTIME_ONLY") ne "yes") {
specialize qw/aom_sad128x128x4d_avg sse2/;
@@ -1201,7 +1195,7 @@
specialize qw/aom_int_pro_col sse2 neon/;
add_proto qw/int aom_vector_var/, "const int16_t *ref, const int16_t *src, const int bwl";
- specialize qw/aom_vector_var neon/;
+ specialize qw/aom_vector_var sse4_1 neon/;
# TODO(kyslov@) bring back SSE2 by extending it to 128 block size
#specialize qw/aom_vector_var neon sse2/;
@@ -1285,9 +1279,9 @@
add_proto qw/unsigned int aom_mse8x8/, "const uint8_t *src_ptr, int source_stride, const uint8_t *ref_ptr, int recon_stride, unsigned int *sse";
specialize qw/aom_mse16x16 sse2 avx2 neon msa/;
- specialize qw/aom_mse16x8 sse2 msa/;
- specialize qw/aom_mse8x16 sse2 msa/;
- specialize qw/aom_mse8x8 sse2 msa/;
+ specialize qw/aom_mse16x8 sse2 neon msa/;
+ specialize qw/aom_mse8x16 sse2 neon msa/;
+ specialize qw/aom_mse8x8 sse2 neon msa/;
if (aom_config("CONFIG_AV1_HIGHBITDEPTH") eq "yes") {
foreach $bd (8, 10, 12) {
@@ -1384,12 +1378,12 @@
specialize qw/aom_sub_pixel_avg_variance4x4 msa sse2 ssse3/;
if (aom_config("CONFIG_REALTIME_ONLY") ne "yes") {
- specialize qw/aom_variance4x16 sse2/;
- specialize qw/aom_variance16x4 sse2 avx2/;
- specialize qw/aom_variance8x32 sse2/;
- specialize qw/aom_variance32x8 sse2 avx2/;
- specialize qw/aom_variance16x64 sse2 avx2/;
- specialize qw/aom_variance64x16 sse2 avx2/;
+ specialize qw/aom_variance4x16 neon sse2/;
+ specialize qw/aom_variance16x4 neon sse2 avx2/;
+ specialize qw/aom_variance8x32 neon sse2/;
+ specialize qw/aom_variance32x8 neon sse2 avx2/;
+ specialize qw/aom_variance16x64 neon sse2 avx2/;
+ specialize qw/aom_variance64x16 neon sse2 avx2/;
specialize qw/aom_sub_pixel_variance4x16 neon sse2 ssse3/;
specialize qw/aom_sub_pixel_variance16x4 neon avx2 sse2 ssse3/;
diff --git a/aom_dsp/aom_simd_inline.h b/aom_dsp/aom_simd_inline.h
index eb333f6..b4b1b35 100644
--- a/aom_dsp/aom_simd_inline.h
+++ b/aom_dsp/aom_simd_inline.h
@@ -18,4 +18,7 @@
#define SIMD_INLINE static AOM_FORCE_INLINE
#endif
+#define SIMD_CLAMP(value, min, max) \
+ ((value) > (max) ? (max) : (value) < (min) ? (min) : (value))
+
#endif // AOM_AOM_DSP_AOM_SIMD_INLINE_H_
diff --git a/aom_dsp/arm/avg_neon.c b/aom_dsp/arm/avg_neon.c
index 9f5b545..593807b 100644
--- a/aom_dsp/arm/avg_neon.c
+++ b/aom_dsp/arm/avg_neon.c
@@ -187,10 +187,11 @@
v_sse = vmlal_s16(v_sse, v_high, v_high);
#endif
}
- int mean = horizontal_add_s32x4(v_mean);
- int sse = horizontal_add_s32x4(v_sse);
+ const int mean = horizontal_add_s32x4(v_mean);
+ const int sse = horizontal_add_s32x4(v_sse);
+ const unsigned int mean_abs = mean >= 0 ? mean : -mean;
// (mean * mean): dynamic range 31 bits.
- int var = sse - ((mean * mean) >> (bwl + 2));
+ const int var = sse - ((mean_abs * mean_abs) >> (bwl + 2));
return var;
}
diff --git a/aom_dsp/arm/highbd_intrapred_neon.c b/aom_dsp/arm/highbd_intrapred_neon.c
index 7544777..fa2f11e 100644
--- a/aom_dsp/arm/highbd_intrapred_neon.c
+++ b/aom_dsp/arm/highbd_intrapred_neon.c
@@ -503,7 +503,7 @@
/* Precompute weighted values that don't vary with |y|. */ \
uint32x4_t weighted_tr_low[(W) >> 3]; \
uint32x4_t weighted_tr_high[(W) >> 3]; \
- for (int i = 0; i<(W)>> 3; ++i) { \
+ for (int i = 0; i < (W) >> 3; ++i) { \
const int x = i << 3; \
const uint16x4_t weights_x_low = \
vld1_u16(smooth_weights_u16 + (W)-4 + x); \
@@ -518,7 +518,7 @@
const uint32x4_t weighted_bl = \
vmull_n_u16(bottom_left_v, 256 - weights_y[y]); \
uint16_t *dst_x = dst; \
- for (int i = 0; i<(W)>> 3; ++i) { \
+ for (int i = 0; i < (W) >> 3; ++i) { \
const int x = i << 3; \
const uint16x4x2_t top_vals = { { vld1_u16(top_row + x), \
vld1_u16(top_row + x + 4) } }; \
@@ -644,7 +644,7 @@
const uint16_t *const weights_y = smooth_weights_u16 + height - 4; \
\
uint16x4x2_t top_vals[(W) >> 3]; \
- for (int i = 0; i<(W)>> 3; ++i) { \
+ for (int i = 0; i < (W) >> 3; ++i) { \
const int x = i << 3; \
top_vals[i].val[0] = vld1_u16(top_row + x); \
top_vals[i].val[1] = vld1_u16(top_row + x + 4); \
@@ -656,7 +656,7 @@
vmull_n_u16(bottom_left_v, 256 - weights_y[y]); \
\
uint16_t *dst_x = dst; \
- for (int i = 0; i<(W)>> 3; ++i) { \
+ for (int i = 0; i < (W) >> 3; ++i) { \
const uint32x4_t weighted_top_low = \
vmlal_n_u16(weighted_bl, top_vals[i].val[0], weights_y[y]); \
vst1_u16(dst_x, \
@@ -776,7 +776,7 @@
uint16x4_t weights_x_high[(W) >> 3]; \
uint32x4_t weighted_tr_low[(W) >> 3]; \
uint32x4_t weighted_tr_high[(W) >> 3]; \
- for (int i = 0; i<(W)>> 3; ++i) { \
+ for (int i = 0; i < (W) >> 3; ++i) { \
const int x = i << 3; \
weights_x_low[i] = vld1_u16(smooth_weights_u16 + (W)-4 + x); \
weighted_tr_low[i] = \
@@ -789,7 +789,7 @@
for (int y = 0; y < height; ++y) { \
uint16_t *dst_x = dst; \
const uint16_t left_y = left_column[y]; \
- for (int i = 0; i<(W)>> 3; ++i) { \
+ for (int i = 0; i < (W) >> 3; ++i) { \
const uint32x4_t weighted_left_low = \
vmlal_n_u16(weighted_tr_low[i], weights_x_low[i], left_y); \
vst1_u16(dst_x, \
diff --git a/aom_dsp/arm/mem_neon.h b/aom_dsp/arm/mem_neon.h
index 81643e9..40be27d 100644
--- a/aom_dsp/arm/mem_neon.h
+++ b/aom_dsp/arm/mem_neon.h
@@ -15,6 +15,64 @@
#include <string.h>
#include "aom_dsp/aom_dsp_common.h"
+// Support for xN Neon intrinsics is lacking in some compilers.
+#if defined(__arm__) || defined(_M_ARM)
+#define ARM_32_BIT
+#endif
+
+// DEFICIENT_CLANG_32_BIT includes clang-cl.
+#if defined(__clang__) && defined(ARM_32_BIT) && \
+ (__clang_major__ <= 6 || (defined(__ANDROID__) && __clang_major__ <= 7))
+#define DEFICIENT_CLANG_32_BIT // This includes clang-cl.
+#endif
+
+#if defined(__GNUC__) && !defined(__clang__) && defined(ARM_32_BIT)
+#define GCC_32_BIT
+#endif
+
+#if defined(DEFICIENT_CLANG_32_BIT) || defined(GCC_32_BIT)
+
+static INLINE uint8x16x3_t vld1q_u8_x3(const uint8_t *ptr) {
+ uint8x16x3_t res = { { vld1q_u8(ptr + 0 * 16), vld1q_u8(ptr + 1 * 16),
+ vld1q_u8(ptr + 2 * 16) } };
+ return res;
+}
+
+static INLINE uint8x16x2_t vld1q_u8_x2(const uint8_t *ptr) {
+ uint8x16x2_t res = { { vld1q_u8(ptr + 0 * 16), vld1q_u8(ptr + 1 * 16) } };
+ return res;
+}
+
+static INLINE uint16x8x4_t vld1q_u16_x4(const uint16_t *ptr) {
+ uint16x8x4_t res = { { vld1q_u16(ptr + 0 * 8), vld1q_u16(ptr + 1 * 8),
+ vld1q_u16(ptr + 2 * 8), vld1q_u16(ptr + 3 * 8) } };
+ return res;
+}
+
+#elif defined(__GNUC__) && !defined(__clang__) // GCC 64-bit.
+#if __GNUC__ < 8
+
+static INLINE uint8x16x2_t vld1q_u8_x2(const uint8_t *ptr) {
+ uint8x16x2_t res = { { vld1q_u8(ptr + 0 * 16), vld1q_u8(ptr + 1 * 16) } };
+ return res;
+}
+
+static INLINE uint16x8x4_t vld1q_u16_x4(const uint16_t *ptr) {
+ uint16x8x4_t res = { { vld1q_u16(ptr + 0 * 8), vld1q_u16(ptr + 1 * 8),
+ vld1q_u16(ptr + 2 * 8), vld1q_u16(ptr + 3 * 8) } };
+ return res;
+}
+#endif // __GNUC__ < 8
+
+#if __GNUC__ < 9
+static INLINE uint8x16x3_t vld1q_u8_x3(const uint8_t *ptr) {
+ uint8x16x3_t res = { { vld1q_u8(ptr + 0 * 16), vld1q_u8(ptr + 1 * 16),
+ vld1q_u8(ptr + 2 * 16) } };
+ return res;
+}
+#endif // __GNUC__ < 9
+#endif // defined(__GNUC__) && !defined(__clang__)
+
static INLINE void store_row2_u8_8x8(uint8_t *s, int p, const uint8x8_t s0,
const uint8x8_t s1) {
vst1_u8(s, s0);
@@ -316,14 +374,25 @@
*s3 = vld1q_s16(s);
}
+// Load 2 sets of 4 bytes when alignment is not guaranteed.
+static INLINE uint8x8_t load_unaligned_u8(const uint8_t *buf, int stride) {
+ uint32_t a;
+ memcpy(&a, buf, 4);
+ buf += stride;
+ uint32x2_t a_u32 = vdup_n_u32(a);
+ memcpy(&a, buf, 4);
+ a_u32 = vset_lane_u32(a, a_u32, 1);
+ return vreinterpret_u8_u32(a_u32);
+}
+
// Load 4 sets of 4 bytes when alignment is not guaranteed.
static INLINE uint8x16_t load_unaligned_u8q(const uint8_t *buf, int stride) {
uint32_t a;
- uint32x4_t a_u32 = vdupq_n_u32(0);
+ uint32x4_t a_u32;
if (stride == 4) return vld1q_u8(buf);
memcpy(&a, buf, 4);
buf += stride;
- a_u32 = vsetq_lane_u32(a, a_u32, 0);
+ a_u32 = vdupq_n_u32(a);
memcpy(&a, buf, 4);
buf += stride;
a_u32 = vsetq_lane_u32(a, a_u32, 1);
@@ -331,7 +400,6 @@
buf += stride;
a_u32 = vsetq_lane_u32(a, a_u32, 2);
memcpy(&a, buf, 4);
- buf += stride;
a_u32 = vsetq_lane_u32(a, a_u32, 3);
return vreinterpretq_u8_u32(a_u32);
}
@@ -343,25 +411,25 @@
memcpy(&a, buf, 4);
buf += stride;
- *tu0 = vset_lane_u32(a, *tu0, 0);
+ *tu0 = vdup_n_u32(a);
memcpy(&a, buf, 4);
buf += stride;
*tu0 = vset_lane_u32(a, *tu0, 1);
memcpy(&a, buf, 4);
buf += stride;
- *tu1 = vset_lane_u32(a, *tu1, 0);
+ *tu1 = vdup_n_u32(a);
memcpy(&a, buf, 4);
buf += stride;
*tu1 = vset_lane_u32(a, *tu1, 1);
memcpy(&a, buf, 4);
buf += stride;
- *tu2 = vset_lane_u32(a, *tu2, 0);
+ *tu2 = vdup_n_u32(a);
memcpy(&a, buf, 4);
buf += stride;
*tu2 = vset_lane_u32(a, *tu2, 1);
memcpy(&a, buf, 4);
buf += stride;
- *tu3 = vset_lane_u32(a, *tu3, 0);
+ *tu3 = vdup_n_u32(a);
memcpy(&a, buf, 4);
*tu3 = vset_lane_u32(a, *tu3, 1);
}
@@ -372,13 +440,13 @@
memcpy(&a, buf, 4);
buf += stride;
- *tu0 = vset_lane_u32(a, *tu0, 0);
+ *tu0 = vdup_n_u32(a);
memcpy(&a, buf, 4);
buf += stride;
*tu0 = vset_lane_u32(a, *tu0, 1);
memcpy(&a, buf, 4);
buf += stride;
- *tu1 = vset_lane_u32(a, *tu1, 0);
+ *tu1 = vdup_n_u32(a);
memcpy(&a, buf, 4);
*tu1 = vset_lane_u32(a, *tu1, 1);
}
@@ -398,9 +466,8 @@
memcpy(&a, buf, 4);
buf += stride;
- *tu0 = vset_lane_u32(a, *tu0, 0);
+ *tu0 = vdup_n_u32(a);
memcpy(&a, buf, 4);
- buf += stride;
*tu0 = vset_lane_u32(a, *tu0, 1);
}
@@ -426,9 +493,8 @@
memcpy(&a, buf, 2);
buf += stride;
- *tu0 = vset_lane_u16(a, *tu0, 0);
+ *tu0 = vdup_n_u16(a);
memcpy(&a, buf, 2);
- buf += stride;
*tu0 = vset_lane_u16(a, *tu0, 1);
}
@@ -472,13 +538,13 @@
memcpy(&a, buf, 8);
buf += stride;
- *tu0 = vsetq_lane_u64(a, *tu0, 0);
+ *tu0 = vdupq_n_u64(a);
memcpy(&a, buf, 8);
buf += stride;
*tu0 = vsetq_lane_u64(a, *tu0, 1);
memcpy(&a, buf, 8);
buf += stride;
- *tu1 = vsetq_lane_u64(a, *tu1, 0);
+ *tu1 = vdupq_n_u64(a);
memcpy(&a, buf, 8);
*tu1 = vsetq_lane_u64(a, *tu1, 1);
}
diff --git a/aom_dsp/arm/sad4d_neon.c b/aom_dsp/arm/sad4d_neon.c
index b62628e..e1eccc3 100644
--- a/aom_dsp/arm/sad4d_neon.c
+++ b/aom_dsp/arm/sad4d_neon.c
@@ -17,550 +17,518 @@
#include "aom/aom_integer.h"
#include "aom_dsp/arm/sum_neon.h"
-// Calculate the absolute difference of 64 bytes from vec_src_00, vec_src_16,
-// vec_src_32, vec_src_48 and ref. Accumulate partial sums in vec_sum_ref_lo
-// and vec_sum_ref_hi.
-static void sad_neon_64(const uint8x16_t vec_src_00,
- const uint8x16_t vec_src_16,
- const uint8x16_t vec_src_32,
- const uint8x16_t vec_src_48, const uint8_t *ref,
- uint16x8_t *vec_sum_ref_lo,
- uint16x8_t *vec_sum_ref_hi) {
- const uint8x16_t vec_ref_00 = vld1q_u8(ref);
- const uint8x16_t vec_ref_16 = vld1q_u8(ref + 16);
- const uint8x16_t vec_ref_32 = vld1q_u8(ref + 32);
- const uint8x16_t vec_ref_48 = vld1q_u8(ref + 48);
+#if defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
- *vec_sum_ref_lo = vabal_u8(*vec_sum_ref_lo, vget_low_u8(vec_src_00),
- vget_low_u8(vec_ref_00));
- *vec_sum_ref_hi = vabal_u8(*vec_sum_ref_hi, vget_high_u8(vec_src_00),
- vget_high_u8(vec_ref_00));
- *vec_sum_ref_lo = vabal_u8(*vec_sum_ref_lo, vget_low_u8(vec_src_16),
- vget_low_u8(vec_ref_16));
- *vec_sum_ref_hi = vabal_u8(*vec_sum_ref_hi, vget_high_u8(vec_src_16),
- vget_high_u8(vec_ref_16));
- *vec_sum_ref_lo = vabal_u8(*vec_sum_ref_lo, vget_low_u8(vec_src_32),
- vget_low_u8(vec_ref_32));
- *vec_sum_ref_hi = vabal_u8(*vec_sum_ref_hi, vget_high_u8(vec_src_32),
- vget_high_u8(vec_ref_32));
- *vec_sum_ref_lo = vabal_u8(*vec_sum_ref_lo, vget_low_u8(vec_src_48),
- vget_low_u8(vec_ref_48));
- *vec_sum_ref_hi = vabal_u8(*vec_sum_ref_hi, vget_high_u8(vec_src_48),
- vget_high_u8(vec_ref_48));
+static INLINE void sad16_neon(uint8x16_t src, uint8x16_t ref,
+ uint32x4_t *const sad_sum) {
+ uint8x16_t abs_diff = vabdq_u8(src, ref);
+ *sad_sum = vdotq_u32(*sad_sum, abs_diff, vdupq_n_u8(1));
}
-// Calculate the absolute difference of 32 bytes from vec_src_00, vec_src_16,
-// and ref. Accumulate partial sums in vec_sum_ref_lo and vec_sum_ref_hi.
-static void sad_neon_32(const uint8x16_t vec_src_00,
- const uint8x16_t vec_src_16, const uint8_t *ref,
- uint16x8_t *vec_sum_ref_lo,
- uint16x8_t *vec_sum_ref_hi) {
- const uint8x16_t vec_ref_00 = vld1q_u8(ref);
- const uint8x16_t vec_ref_16 = vld1q_u8(ref + 16);
+static INLINE void sad128xhx4d_neon(const uint8_t *src, int src_stride,
+ const uint8_t *const ref[4], int ref_stride,
+ uint32_t res[4], int h) {
+ uint32x4_t sum_lo[4] = { vdupq_n_u32(0), vdupq_n_u32(0), vdupq_n_u32(0),
+ vdupq_n_u32(0) };
+ uint32x4_t sum_hi[4] = { vdupq_n_u32(0), vdupq_n_u32(0), vdupq_n_u32(0),
+ vdupq_n_u32(0) };
- *vec_sum_ref_lo = vabal_u8(*vec_sum_ref_lo, vget_low_u8(vec_src_00),
- vget_low_u8(vec_ref_00));
- *vec_sum_ref_hi = vabal_u8(*vec_sum_ref_hi, vget_high_u8(vec_src_00),
- vget_high_u8(vec_ref_00));
- *vec_sum_ref_lo = vabal_u8(*vec_sum_ref_lo, vget_low_u8(vec_src_16),
- vget_low_u8(vec_ref_16));
- *vec_sum_ref_hi = vabal_u8(*vec_sum_ref_hi, vget_high_u8(vec_src_16),
- vget_high_u8(vec_ref_16));
+ int i = 0;
+ do {
+ const uint8x16_t s0 = vld1q_u8(src + i * src_stride);
+ sad16_neon(s0, vld1q_u8(ref[0] + i * ref_stride), &sum_lo[0]);
+ sad16_neon(s0, vld1q_u8(ref[1] + i * ref_stride), &sum_lo[1]);
+ sad16_neon(s0, vld1q_u8(ref[2] + i * ref_stride), &sum_lo[2]);
+ sad16_neon(s0, vld1q_u8(ref[3] + i * ref_stride), &sum_lo[3]);
+
+ const uint8x16_t s1 = vld1q_u8(src + i * src_stride + 16);
+ sad16_neon(s1, vld1q_u8(ref[0] + i * ref_stride + 16), &sum_hi[0]);
+ sad16_neon(s1, vld1q_u8(ref[1] + i * ref_stride + 16), &sum_hi[1]);
+ sad16_neon(s1, vld1q_u8(ref[2] + i * ref_stride + 16), &sum_hi[2]);
+ sad16_neon(s1, vld1q_u8(ref[3] + i * ref_stride + 16), &sum_hi[3]);
+
+ const uint8x16_t s2 = vld1q_u8(src + i * src_stride + 32);
+ sad16_neon(s2, vld1q_u8(ref[0] + i * ref_stride + 32), &sum_lo[0]);
+ sad16_neon(s2, vld1q_u8(ref[1] + i * ref_stride + 32), &sum_lo[1]);
+ sad16_neon(s2, vld1q_u8(ref[2] + i * ref_stride + 32), &sum_lo[2]);
+ sad16_neon(s2, vld1q_u8(ref[3] + i * ref_stride + 32), &sum_lo[3]);
+
+ const uint8x16_t s3 = vld1q_u8(src + i * src_stride + 48);
+ sad16_neon(s3, vld1q_u8(ref[0] + i * ref_stride + 48), &sum_hi[0]);
+ sad16_neon(s3, vld1q_u8(ref[1] + i * ref_stride + 48), &sum_hi[1]);
+ sad16_neon(s3, vld1q_u8(ref[2] + i * ref_stride + 48), &sum_hi[2]);
+ sad16_neon(s3, vld1q_u8(ref[3] + i * ref_stride + 48), &sum_hi[3]);
+
+ const uint8x16_t s4 = vld1q_u8(src + i * src_stride + 64);
+ sad16_neon(s4, vld1q_u8(ref[0] + i * ref_stride + 64), &sum_lo[0]);
+ sad16_neon(s4, vld1q_u8(ref[1] + i * ref_stride + 64), &sum_lo[1]);
+ sad16_neon(s4, vld1q_u8(ref[2] + i * ref_stride + 64), &sum_lo[2]);
+ sad16_neon(s4, vld1q_u8(ref[3] + i * ref_stride + 64), &sum_lo[3]);
+
+ const uint8x16_t s5 = vld1q_u8(src + i * src_stride + 80);
+ sad16_neon(s5, vld1q_u8(ref[0] + i * ref_stride + 80), &sum_hi[0]);
+ sad16_neon(s5, vld1q_u8(ref[1] + i * ref_stride + 80), &sum_hi[1]);
+ sad16_neon(s5, vld1q_u8(ref[2] + i * ref_stride + 80), &sum_hi[2]);
+ sad16_neon(s5, vld1q_u8(ref[3] + i * ref_stride + 80), &sum_hi[3]);
+
+ const uint8x16_t s6 = vld1q_u8(src + i * src_stride + 96);
+ sad16_neon(s6, vld1q_u8(ref[0] + i * ref_stride + 96), &sum_lo[0]);
+ sad16_neon(s6, vld1q_u8(ref[1] + i * ref_stride + 96), &sum_lo[1]);
+ sad16_neon(s6, vld1q_u8(ref[2] + i * ref_stride + 96), &sum_lo[2]);
+ sad16_neon(s6, vld1q_u8(ref[3] + i * ref_stride + 96), &sum_lo[3]);
+
+ const uint8x16_t s7 = vld1q_u8(src + i * src_stride + 112);
+ sad16_neon(s7, vld1q_u8(ref[0] + i * ref_stride + 112), &sum_hi[0]);
+ sad16_neon(s7, vld1q_u8(ref[1] + i * ref_stride + 112), &sum_hi[1]);
+ sad16_neon(s7, vld1q_u8(ref[2] + i * ref_stride + 112), &sum_hi[2]);
+ sad16_neon(s7, vld1q_u8(ref[3] + i * ref_stride + 112), &sum_hi[3]);
+
+ i++;
+ } while (i < h);
+
+ uint32x4_t res0 = vpaddq_u32(vaddq_u32(sum_lo[0], sum_hi[0]),
+ vaddq_u32(sum_lo[1], sum_hi[1]));
+ uint32x4_t res1 = vpaddq_u32(vaddq_u32(sum_lo[2], sum_hi[2]),
+ vaddq_u32(sum_lo[3], sum_hi[3]));
+ vst1q_u32(res, vpaddq_u32(res0, res1));
}
-void aom_sad64x64x4d_neon(const uint8_t *src, int src_stride,
- const uint8_t *const ref[4], int ref_stride,
- uint32_t res[4]) {
- int i;
- uint16x8_t vec_sum_ref0_lo = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref0_hi = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref1_lo = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref1_hi = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref2_lo = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref2_hi = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref3_lo = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref3_hi = vdupq_n_u16(0);
- const uint8_t *ref0, *ref1, *ref2, *ref3;
- ref0 = ref[0];
- ref1 = ref[1];
- ref2 = ref[2];
- ref3 = ref[3];
+static INLINE void sad64xhx4d_neon(const uint8_t *src, int src_stride,
+ const uint8_t *const ref[4], int ref_stride,
+ uint32_t res[4], int h) {
+ uint32x4_t sum_lo[4] = { vdupq_n_u32(0), vdupq_n_u32(0), vdupq_n_u32(0),
+ vdupq_n_u32(0) };
+ uint32x4_t sum_hi[4] = { vdupq_n_u32(0), vdupq_n_u32(0), vdupq_n_u32(0),
+ vdupq_n_u32(0) };
- for (i = 0; i < 64; ++i) {
- const uint8x16_t vec_src_00 = vld1q_u8(src);
- const uint8x16_t vec_src_16 = vld1q_u8(src + 16);
- const uint8x16_t vec_src_32 = vld1q_u8(src + 32);
- const uint8x16_t vec_src_48 = vld1q_u8(src + 48);
+ int i = 0;
+ do {
+ const uint8x16_t s0 = vld1q_u8(src + i * src_stride);
+ sad16_neon(s0, vld1q_u8(ref[0] + i * ref_stride), &sum_lo[0]);
+ sad16_neon(s0, vld1q_u8(ref[1] + i * ref_stride), &sum_lo[1]);
+ sad16_neon(s0, vld1q_u8(ref[2] + i * ref_stride), &sum_lo[2]);
+ sad16_neon(s0, vld1q_u8(ref[3] + i * ref_stride), &sum_lo[3]);
- sad_neon_64(vec_src_00, vec_src_16, vec_src_32, vec_src_48, ref0,
- &vec_sum_ref0_lo, &vec_sum_ref0_hi);
- sad_neon_64(vec_src_00, vec_src_16, vec_src_32, vec_src_48, ref1,
- &vec_sum_ref1_lo, &vec_sum_ref1_hi);
- sad_neon_64(vec_src_00, vec_src_16, vec_src_32, vec_src_48, ref2,
- &vec_sum_ref2_lo, &vec_sum_ref2_hi);
- sad_neon_64(vec_src_00, vec_src_16, vec_src_32, vec_src_48, ref3,
- &vec_sum_ref3_lo, &vec_sum_ref3_hi);
+ const uint8x16_t s1 = vld1q_u8(src + i * src_stride + 16);
+ sad16_neon(s1, vld1q_u8(ref[0] + i * ref_stride + 16), &sum_hi[0]);
+ sad16_neon(s1, vld1q_u8(ref[1] + i * ref_stride + 16), &sum_hi[1]);
+ sad16_neon(s1, vld1q_u8(ref[2] + i * ref_stride + 16), &sum_hi[2]);
+ sad16_neon(s1, vld1q_u8(ref[3] + i * ref_stride + 16), &sum_hi[3]);
- src += src_stride;
- ref0 += ref_stride;
- ref1 += ref_stride;
- ref2 += ref_stride;
- ref3 += ref_stride;
+ const uint8x16_t s2 = vld1q_u8(src + i * src_stride + 32);
+ sad16_neon(s2, vld1q_u8(ref[0] + i * ref_stride + 32), &sum_lo[0]);
+ sad16_neon(s2, vld1q_u8(ref[1] + i * ref_stride + 32), &sum_lo[1]);
+ sad16_neon(s2, vld1q_u8(ref[2] + i * ref_stride + 32), &sum_lo[2]);
+ sad16_neon(s2, vld1q_u8(ref[3] + i * ref_stride + 32), &sum_lo[3]);
+
+ const uint8x16_t s3 = vld1q_u8(src + i * src_stride + 48);
+ sad16_neon(s3, vld1q_u8(ref[0] + i * ref_stride + 48), &sum_hi[0]);
+ sad16_neon(s3, vld1q_u8(ref[1] + i * ref_stride + 48), &sum_hi[1]);
+ sad16_neon(s3, vld1q_u8(ref[2] + i * ref_stride + 48), &sum_hi[2]);
+ sad16_neon(s3, vld1q_u8(ref[3] + i * ref_stride + 48), &sum_hi[3]);
+
+ i++;
+ } while (i < h);
+
+ uint32x4_t res0 = vpaddq_u32(vaddq_u32(sum_lo[0], sum_hi[0]),
+ vaddq_u32(sum_lo[1], sum_hi[1]));
+ uint32x4_t res1 = vpaddq_u32(vaddq_u32(sum_lo[2], sum_hi[2]),
+ vaddq_u32(sum_lo[3], sum_hi[3]));
+ vst1q_u32(res, vpaddq_u32(res0, res1));
+}
+
+static INLINE void sad32xhx4d_neon(const uint8_t *src, int src_stride,
+ const uint8_t *const ref[4], int ref_stride,
+ uint32_t res[4], int h) {
+ uint32x4_t sum_lo[4] = { vdupq_n_u32(0), vdupq_n_u32(0), vdupq_n_u32(0),
+ vdupq_n_u32(0) };
+ uint32x4_t sum_hi[4] = { vdupq_n_u32(0), vdupq_n_u32(0), vdupq_n_u32(0),
+ vdupq_n_u32(0) };
+
+ int i = 0;
+ do {
+ const uint8x16_t s0 = vld1q_u8(src + i * src_stride);
+ sad16_neon(s0, vld1q_u8(ref[0] + i * ref_stride), &sum_lo[0]);
+ sad16_neon(s0, vld1q_u8(ref[1] + i * ref_stride), &sum_lo[1]);
+ sad16_neon(s0, vld1q_u8(ref[2] + i * ref_stride), &sum_lo[2]);
+ sad16_neon(s0, vld1q_u8(ref[3] + i * ref_stride), &sum_lo[3]);
+
+ const uint8x16_t s1 = vld1q_u8(src + i * src_stride + 16);
+ sad16_neon(s1, vld1q_u8(ref[0] + i * ref_stride + 16), &sum_hi[0]);
+ sad16_neon(s1, vld1q_u8(ref[1] + i * ref_stride + 16), &sum_hi[1]);
+ sad16_neon(s1, vld1q_u8(ref[2] + i * ref_stride + 16), &sum_hi[2]);
+ sad16_neon(s1, vld1q_u8(ref[3] + i * ref_stride + 16), &sum_hi[3]);
+
+ i++;
+ } while (i < h);
+
+ uint32x4_t res0 = vpaddq_u32(vaddq_u32(sum_lo[0], sum_hi[0]),
+ vaddq_u32(sum_lo[1], sum_hi[1]));
+ uint32x4_t res1 = vpaddq_u32(vaddq_u32(sum_lo[2], sum_hi[2]),
+ vaddq_u32(sum_lo[3], sum_hi[3]));
+ vst1q_u32(res, vpaddq_u32(res0, res1));
+}
+
+static INLINE void sad16xhx4d_neon(const uint8_t *src, int src_stride,
+ const uint8_t *const ref[4], int ref_stride,
+ uint32_t res[4], int h) {
+ uint32x4_t sum[4] = { vdupq_n_u32(0), vdupq_n_u32(0), vdupq_n_u32(0),
+ vdupq_n_u32(0) };
+
+ int i = 0;
+ do {
+ const uint8x16_t s = vld1q_u8(src + i * src_stride);
+ sad16_neon(s, vld1q_u8(ref[0] + i * ref_stride), &sum[0]);
+ sad16_neon(s, vld1q_u8(ref[1] + i * ref_stride), &sum[1]);
+ sad16_neon(s, vld1q_u8(ref[2] + i * ref_stride), &sum[2]);
+ sad16_neon(s, vld1q_u8(ref[3] + i * ref_stride), &sum[3]);
+
+ i++;
+ } while (i < h);
+
+ uint32x4_t res0 = vpaddq_u32(sum[0], sum[1]);
+ uint32x4_t res1 = vpaddq_u32(sum[2], sum[3]);
+ vst1q_u32(res, vpaddq_u32(res0, res1));
+}
+
+#else // !(defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD))
+
+static INLINE void sad16_neon(uint8x16_t src, uint8x16_t ref,
+ uint16x8_t *const sad_sum) {
+ uint8x16_t abs_diff = vabdq_u8(src, ref);
+ *sad_sum = vpadalq_u8(*sad_sum, abs_diff);
+}
+
+static INLINE void sad128xhx4d_neon(const uint8_t *src, int src_stride,
+ const uint8_t *const ref[4], int ref_stride,
+ uint32_t res[4], int h) {
+ vst1q_u32(res, vdupq_n_u32(0));
+ int h_tmp = h > 32 ? 32 : h;
+
+ int i = 0;
+ do {
+ uint16x8_t sum_lo[4] = { vdupq_n_u16(0), vdupq_n_u16(0), vdupq_n_u16(0),
+ vdupq_n_u16(0) };
+ uint16x8_t sum_hi[4] = { vdupq_n_u16(0), vdupq_n_u16(0), vdupq_n_u16(0),
+ vdupq_n_u16(0) };
+
+ do {
+ const uint8x16_t s0 = vld1q_u8(src + i * src_stride);
+ sad16_neon(s0, vld1q_u8(ref[0] + i * ref_stride), &sum_lo[0]);
+ sad16_neon(s0, vld1q_u8(ref[1] + i * ref_stride), &sum_lo[1]);
+ sad16_neon(s0, vld1q_u8(ref[2] + i * ref_stride), &sum_lo[2]);
+ sad16_neon(s0, vld1q_u8(ref[3] + i * ref_stride), &sum_lo[3]);
+
+ const uint8x16_t s1 = vld1q_u8(src + i * src_stride + 16);
+ sad16_neon(s1, vld1q_u8(ref[0] + i * ref_stride + 16), &sum_hi[0]);
+ sad16_neon(s1, vld1q_u8(ref[1] + i * ref_stride + 16), &sum_hi[1]);
+ sad16_neon(s1, vld1q_u8(ref[2] + i * ref_stride + 16), &sum_hi[2]);
+ sad16_neon(s1, vld1q_u8(ref[3] + i * ref_stride + 16), &sum_hi[3]);
+
+ const uint8x16_t s2 = vld1q_u8(src + i * src_stride + 32);
+ sad16_neon(s2, vld1q_u8(ref[0] + i * ref_stride + 32), &sum_lo[0]);
+ sad16_neon(s2, vld1q_u8(ref[1] + i * ref_stride + 32), &sum_lo[1]);
+ sad16_neon(s2, vld1q_u8(ref[2] + i * ref_stride + 32), &sum_lo[2]);
+ sad16_neon(s2, vld1q_u8(ref[3] + i * ref_stride + 32), &sum_lo[3]);
+
+ const uint8x16_t s3 = vld1q_u8(src + i * src_stride + 48);
+ sad16_neon(s3, vld1q_u8(ref[0] + i * ref_stride + 48), &sum_hi[0]);
+ sad16_neon(s3, vld1q_u8(ref[1] + i * ref_stride + 48), &sum_hi[1]);
+ sad16_neon(s3, vld1q_u8(ref[2] + i * ref_stride + 48), &sum_hi[2]);
+ sad16_neon(s3, vld1q_u8(ref[3] + i * ref_stride + 48), &sum_hi[3]);
+
+ const uint8x16_t s4 = vld1q_u8(src + i * src_stride + 64);
+ sad16_neon(s4, vld1q_u8(ref[0] + i * ref_stride + 64), &sum_lo[0]);
+ sad16_neon(s4, vld1q_u8(ref[1] + i * ref_stride + 64), &sum_lo[1]);
+ sad16_neon(s4, vld1q_u8(ref[2] + i * ref_stride + 64), &sum_lo[2]);
+ sad16_neon(s4, vld1q_u8(ref[3] + i * ref_stride + 64), &sum_lo[3]);
+
+ const uint8x16_t s5 = vld1q_u8(src + i * src_stride + 80);
+ sad16_neon(s5, vld1q_u8(ref[0] + i * ref_stride + 80), &sum_hi[0]);
+ sad16_neon(s5, vld1q_u8(ref[1] + i * ref_stride + 80), &sum_hi[1]);
+ sad16_neon(s5, vld1q_u8(ref[2] + i * ref_stride + 80), &sum_hi[2]);
+ sad16_neon(s5, vld1q_u8(ref[3] + i * ref_stride + 80), &sum_hi[3]);
+
+ const uint8x16_t s6 = vld1q_u8(src + i * src_stride + 96);
+ sad16_neon(s6, vld1q_u8(ref[0] + i * ref_stride + 96), &sum_lo[0]);
+ sad16_neon(s6, vld1q_u8(ref[1] + i * ref_stride + 96), &sum_lo[1]);
+ sad16_neon(s6, vld1q_u8(ref[2] + i * ref_stride + 96), &sum_lo[2]);
+ sad16_neon(s6, vld1q_u8(ref[3] + i * ref_stride + 96), &sum_lo[3]);
+
+ const uint8x16_t s7 = vld1q_u8(src + i * src_stride + 112);
+ sad16_neon(s7, vld1q_u8(ref[0] + i * ref_stride + 112), &sum_hi[0]);
+ sad16_neon(s7, vld1q_u8(ref[1] + i * ref_stride + 112), &sum_hi[1]);
+ sad16_neon(s7, vld1q_u8(ref[2] + i * ref_stride + 112), &sum_hi[2]);
+ sad16_neon(s7, vld1q_u8(ref[3] + i * ref_stride + 112), &sum_hi[3]);
+
+ i++;
+ } while (i < h_tmp);
+
+ res[0] += horizontal_long_add_u16x8(sum_lo[0], sum_hi[0]);
+ res[1] += horizontal_long_add_u16x8(sum_lo[1], sum_hi[1]);
+ res[2] += horizontal_long_add_u16x8(sum_lo[2], sum_hi[2]);
+ res[3] += horizontal_long_add_u16x8(sum_lo[3], sum_hi[3]);
+
+ h_tmp += 32;
+ } while (i < h);
+}
+
+static INLINE void sad64xhx4d_neon(const uint8_t *src, int src_stride,
+ const uint8_t *const ref[4], int ref_stride,
+ uint32_t res[4], int h) {
+ vst1q_u32(res, vdupq_n_u32(0));
+ int h_tmp = h > 64 ? 64 : h;
+
+ int i = 0;
+ do {
+ uint16x8_t sum_lo[4] = { vdupq_n_u16(0), vdupq_n_u16(0), vdupq_n_u16(0),
+ vdupq_n_u16(0) };
+ uint16x8_t sum_hi[4] = { vdupq_n_u16(0), vdupq_n_u16(0), vdupq_n_u16(0),
+ vdupq_n_u16(0) };
+
+ do {
+ const uint8x16_t s0 = vld1q_u8(src + i * src_stride);
+ sad16_neon(s0, vld1q_u8(ref[0] + i * ref_stride), &sum_lo[0]);
+ sad16_neon(s0, vld1q_u8(ref[1] + i * ref_stride), &sum_lo[1]);
+ sad16_neon(s0, vld1q_u8(ref[2] + i * ref_stride), &sum_lo[2]);
+ sad16_neon(s0, vld1q_u8(ref[3] + i * ref_stride), &sum_lo[3]);
+
+ const uint8x16_t s1 = vld1q_u8(src + i * src_stride + 16);
+ sad16_neon(s1, vld1q_u8(ref[0] + i * ref_stride + 16), &sum_hi[0]);
+ sad16_neon(s1, vld1q_u8(ref[1] + i * ref_stride + 16), &sum_hi[1]);
+ sad16_neon(s1, vld1q_u8(ref[2] + i * ref_stride + 16), &sum_hi[2]);
+ sad16_neon(s1, vld1q_u8(ref[3] + i * ref_stride + 16), &sum_hi[3]);
+
+ const uint8x16_t s2 = vld1q_u8(src + i * src_stride + 32);
+ sad16_neon(s2, vld1q_u8(ref[0] + i * ref_stride + 32), &sum_lo[0]);
+ sad16_neon(s2, vld1q_u8(ref[1] + i * ref_stride + 32), &sum_lo[1]);
+ sad16_neon(s2, vld1q_u8(ref[2] + i * ref_stride + 32), &sum_lo[2]);
+ sad16_neon(s2, vld1q_u8(ref[3] + i * ref_stride + 32), &sum_lo[3]);
+
+ const uint8x16_t s3 = vld1q_u8(src + i * src_stride + 48);
+ sad16_neon(s3, vld1q_u8(ref[0] + i * ref_stride + 48), &sum_hi[0]);
+ sad16_neon(s3, vld1q_u8(ref[1] + i * ref_stride + 48), &sum_hi[1]);
+ sad16_neon(s3, vld1q_u8(ref[2] + i * ref_stride + 48), &sum_hi[2]);
+ sad16_neon(s3, vld1q_u8(ref[3] + i * ref_stride + 48), &sum_hi[3]);
+
+ i++;
+ } while (i < h_tmp);
+
+ res[0] += horizontal_long_add_u16x8(sum_lo[0], sum_hi[0]);
+ res[1] += horizontal_long_add_u16x8(sum_lo[1], sum_hi[1]);
+ res[2] += horizontal_long_add_u16x8(sum_lo[2], sum_hi[2]);
+ res[3] += horizontal_long_add_u16x8(sum_lo[3], sum_hi[3]);
+
+ h_tmp += 64;
+ } while (i < h);
+}
+
+static INLINE void sad32xhx4d_neon(const uint8_t *src, int src_stride,
+ const uint8_t *const ref[4], int ref_stride,
+ uint32_t res[4], int h) {
+ uint16x8_t sum_lo[4] = { vdupq_n_u16(0), vdupq_n_u16(0), vdupq_n_u16(0),
+ vdupq_n_u16(0) };
+ uint16x8_t sum_hi[4] = { vdupq_n_u16(0), vdupq_n_u16(0), vdupq_n_u16(0),
+ vdupq_n_u16(0) };
+
+ int i = 0;
+ do {
+ const uint8x16_t s0 = vld1q_u8(src + i * src_stride);
+ sad16_neon(s0, vld1q_u8(ref[0] + i * ref_stride), &sum_lo[0]);
+ sad16_neon(s0, vld1q_u8(ref[1] + i * ref_stride), &sum_lo[1]);
+ sad16_neon(s0, vld1q_u8(ref[2] + i * ref_stride), &sum_lo[2]);
+ sad16_neon(s0, vld1q_u8(ref[3] + i * ref_stride), &sum_lo[3]);
+
+ const uint8x16_t s1 = vld1q_u8(src + i * src_stride + 16);
+ sad16_neon(s1, vld1q_u8(ref[0] + i * ref_stride + 16), &sum_hi[0]);
+ sad16_neon(s1, vld1q_u8(ref[1] + i * ref_stride + 16), &sum_hi[1]);
+ sad16_neon(s1, vld1q_u8(ref[2] + i * ref_stride + 16), &sum_hi[2]);
+ sad16_neon(s1, vld1q_u8(ref[3] + i * ref_stride + 16), &sum_hi[3]);
+
+ i++;
+ } while (i < h);
+
+ res[0] = horizontal_long_add_u16x8(sum_lo[0], sum_hi[0]);
+ res[1] = horizontal_long_add_u16x8(sum_lo[1], sum_hi[1]);
+ res[2] = horizontal_long_add_u16x8(sum_lo[2], sum_hi[2]);
+ res[3] = horizontal_long_add_u16x8(sum_lo[3], sum_hi[3]);
+}
+
+static INLINE void sad16xhx4d_neon(const uint8_t *src, int src_stride,
+ const uint8_t *const ref[4], int ref_stride,
+ uint32_t res[4], int h) {
+ uint16x8_t sum[4] = { vdupq_n_u16(0), vdupq_n_u16(0), vdupq_n_u16(0),
+ vdupq_n_u16(0) };
+
+ int i = 0;
+ do {
+ const uint8x16_t s = vld1q_u8(src + i * src_stride);
+ sad16_neon(s, vld1q_u8(ref[0] + i * ref_stride), &sum[0]);
+ sad16_neon(s, vld1q_u8(ref[1] + i * ref_stride), &sum[1]);
+ sad16_neon(s, vld1q_u8(ref[2] + i * ref_stride), &sum[2]);
+ sad16_neon(s, vld1q_u8(ref[3] + i * ref_stride), &sum[3]);
+
+ i++;
+ } while (i < h);
+
+ res[0] = horizontal_add_u16x8(sum[0]);
+ res[1] = horizontal_add_u16x8(sum[1]);
+ res[2] = horizontal_add_u16x8(sum[2]);
+ res[3] = horizontal_add_u16x8(sum[3]);
+}
+
+#endif // defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+
+static INLINE void sad8_neon(uint8x8_t src, uint8x8_t ref,
+ uint16x8_t *const sad_sum) {
+ uint8x8_t abs_diff = vabd_u8(src, ref);
+ *sad_sum = vaddw_u8(*sad_sum, abs_diff);
+}
+
+static INLINE void sad8xhx4d_neon(const uint8_t *src, int src_stride,
+ const uint8_t *const ref[4], int ref_stride,
+ uint32_t res[4], int h) {
+ uint16x8_t sum[4] = { vdupq_n_u16(0), vdupq_n_u16(0), vdupq_n_u16(0),
+ vdupq_n_u16(0) };
+
+ int i = 0;
+ do {
+ const uint8x8_t s = vld1_u8(src + i * src_stride);
+ sad8_neon(s, vld1_u8(ref[0] + i * ref_stride), &sum[0]);
+ sad8_neon(s, vld1_u8(ref[1] + i * ref_stride), &sum[1]);
+ sad8_neon(s, vld1_u8(ref[2] + i * ref_stride), &sum[2]);
+ sad8_neon(s, vld1_u8(ref[3] + i * ref_stride), &sum[3]);
+
+ i++;
+ } while (i < h);
+
+ res[0] = horizontal_add_u16x8(sum[0]);
+ res[1] = horizontal_add_u16x8(sum[1]);
+ res[2] = horizontal_add_u16x8(sum[2]);
+ res[3] = horizontal_add_u16x8(sum[3]);
+}
+
+static INLINE void sad4xhx4d_neon(const uint8_t *src, int src_stride,
+ const uint8_t *const ref[4], int ref_stride,
+ uint32_t res[4], int h) {
+ uint16x8_t sum[4] = { vdupq_n_u16(0), vdupq_n_u16(0), vdupq_n_u16(0),
+ vdupq_n_u16(0) };
+
+ int i = 0;
+ do {
+ uint32x2_t s, r0, r1, r2, r3;
+ uint32_t s_lo, s_hi, r0_lo, r0_hi, r1_lo, r1_hi, r2_lo, r2_hi, r3_lo, r3_hi;
+
+ memcpy(&s_lo, src + i * src_stride, 4);
+ memcpy(&r0_lo, ref[0] + i * ref_stride, 4);
+ memcpy(&r1_lo, ref[1] + i * ref_stride, 4);
+ memcpy(&r2_lo, ref[2] + i * ref_stride, 4);
+ memcpy(&r3_lo, ref[3] + i * ref_stride, 4);
+ s = vdup_n_u32(s_lo);
+ r0 = vdup_n_u32(r0_lo);
+ r1 = vdup_n_u32(r1_lo);
+ r2 = vdup_n_u32(r2_lo);
+ r3 = vdup_n_u32(r3_lo);
+
+ memcpy(&s_hi, src + (i + 1) * src_stride, 4);
+ memcpy(&r0_hi, ref[0] + (i + 1) * ref_stride, 4);
+ memcpy(&r1_hi, ref[1] + (i + 1) * ref_stride, 4);
+ memcpy(&r2_hi, ref[2] + (i + 1) * ref_stride, 4);
+ memcpy(&r3_hi, ref[3] + (i + 1) * ref_stride, 4);
+ s = vset_lane_u32(s_hi, s, 1);
+ r0 = vset_lane_u32(r0_hi, r0, 1);
+ r1 = vset_lane_u32(r1_hi, r1, 1);
+ r2 = vset_lane_u32(r2_hi, r2, 1);
+ r3 = vset_lane_u32(r3_hi, r3, 1);
+
+ sad8_neon(vreinterpret_u8_u32(s), vreinterpret_u8_u32(r0), &sum[0]);
+ sad8_neon(vreinterpret_u8_u32(s), vreinterpret_u8_u32(r1), &sum[1]);
+ sad8_neon(vreinterpret_u8_u32(s), vreinterpret_u8_u32(r2), &sum[2]);
+ sad8_neon(vreinterpret_u8_u32(s), vreinterpret_u8_u32(r3), &sum[3]);
+
+ i += 2;
+ } while (i < h);
+
+ res[0] = horizontal_add_u16x8(sum[0]);
+ res[1] = horizontal_add_u16x8(sum[1]);
+ res[2] = horizontal_add_u16x8(sum[2]);
+ res[3] = horizontal_add_u16x8(sum[3]);
+}
+
+#define SAD_WXH_4D_NEON(w, h) \
+ void aom_sad##w##x##h##x4d_neon(const uint8_t *src, int src_stride, \
+ const uint8_t *const ref[4], int ref_stride, \
+ uint32_t res[4]) { \
+ sad##w##xhx4d_neon(src, src_stride, ref, ref_stride, res, (h)); \
}
- res[0] = horizontal_long_add_u16x8(vec_sum_ref0_lo, vec_sum_ref0_hi);
- res[1] = horizontal_long_add_u16x8(vec_sum_ref1_lo, vec_sum_ref1_hi);
- res[2] = horizontal_long_add_u16x8(vec_sum_ref2_lo, vec_sum_ref2_hi);
- res[3] = horizontal_long_add_u16x8(vec_sum_ref3_lo, vec_sum_ref3_hi);
-}
+SAD_WXH_4D_NEON(4, 4)
+SAD_WXH_4D_NEON(4, 8)
+SAD_WXH_4D_NEON(4, 16)
+SAD_WXH_4D_NEON(4, 32)
-void aom_sad32x32x4d_neon(const uint8_t *src, int src_stride,
- const uint8_t *const ref[4], int ref_stride,
- uint32_t res[4]) {
- int i;
- uint16x8_t vec_sum_ref0_lo = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref0_hi = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref1_lo = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref1_hi = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref2_lo = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref2_hi = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref3_lo = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref3_hi = vdupq_n_u16(0);
- const uint8_t *ref0, *ref1, *ref2, *ref3;
- ref0 = ref[0];
- ref1 = ref[1];
- ref2 = ref[2];
- ref3 = ref[3];
+SAD_WXH_4D_NEON(8, 4)
+SAD_WXH_4D_NEON(8, 8)
+SAD_WXH_4D_NEON(8, 16)
+SAD_WXH_4D_NEON(8, 32)
- for (i = 0; i < 32; ++i) {
- const uint8x16_t vec_src_00 = vld1q_u8(src);
- const uint8x16_t vec_src_16 = vld1q_u8(src + 16);
+SAD_WXH_4D_NEON(16, 4)
+SAD_WXH_4D_NEON(16, 8)
+SAD_WXH_4D_NEON(16, 16)
+SAD_WXH_4D_NEON(16, 32)
+SAD_WXH_4D_NEON(16, 64)
- sad_neon_32(vec_src_00, vec_src_16, ref0, &vec_sum_ref0_lo,
- &vec_sum_ref0_hi);
- sad_neon_32(vec_src_00, vec_src_16, ref1, &vec_sum_ref1_lo,
- &vec_sum_ref1_hi);
- sad_neon_32(vec_src_00, vec_src_16, ref2, &vec_sum_ref2_lo,
- &vec_sum_ref2_hi);
- sad_neon_32(vec_src_00, vec_src_16, ref3, &vec_sum_ref3_lo,
- &vec_sum_ref3_hi);
+SAD_WXH_4D_NEON(32, 8)
+SAD_WXH_4D_NEON(32, 16)
+SAD_WXH_4D_NEON(32, 32)
+SAD_WXH_4D_NEON(32, 64)
- src += src_stride;
- ref0 += ref_stride;
- ref1 += ref_stride;
- ref2 += ref_stride;
- ref3 += ref_stride;
- }
+SAD_WXH_4D_NEON(64, 16)
+SAD_WXH_4D_NEON(64, 32)
+SAD_WXH_4D_NEON(64, 64)
+SAD_WXH_4D_NEON(64, 128)
- res[0] = horizontal_long_add_u16x8(vec_sum_ref0_lo, vec_sum_ref0_hi);
- res[1] = horizontal_long_add_u16x8(vec_sum_ref1_lo, vec_sum_ref1_hi);
- res[2] = horizontal_long_add_u16x8(vec_sum_ref2_lo, vec_sum_ref2_hi);
- res[3] = horizontal_long_add_u16x8(vec_sum_ref3_lo, vec_sum_ref3_hi);
-}
+SAD_WXH_4D_NEON(128, 64)
+SAD_WXH_4D_NEON(128, 128)
-void aom_sad16x16x4d_neon(const uint8_t *src, int src_stride,
- const uint8_t *const ref[4], int ref_stride,
- uint32_t res[4]) {
- int i;
- uint16x8_t vec_sum_ref0_lo = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref0_hi = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref1_lo = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref1_hi = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref2_lo = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref2_hi = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref3_lo = vdupq_n_u16(0);
- uint16x8_t vec_sum_ref3_hi = vdupq_n_u16(0);
- const uint8_t *ref0, *ref1, *ref2, *ref3;
- ref0 = ref[0];
- ref1 = ref[1];
- ref2 = ref[2];
- ref3 = ref[3];
+#undef SAD_WXH_4D_NEON
- for (i = 0; i < 16; ++i) {
- const uint8x16_t vec_src = vld1q_u8(src);
- const uint8x16_t vec_ref0 = vld1q_u8(ref0);
- const uint8x16_t vec_ref1 = vld1q_u8(ref1);
- const uint8x16_t vec_ref2 = vld1q_u8(ref2);
- const uint8x16_t vec_ref3 = vld1q_u8(ref3);
-
- vec_sum_ref0_lo =
- vabal_u8(vec_sum_ref0_lo, vget_low_u8(vec_src), vget_low_u8(vec_ref0));
- vec_sum_ref0_hi = vabal_u8(vec_sum_ref0_hi, vget_high_u8(vec_src),
- vget_high_u8(vec_ref0));
- vec_sum_ref1_lo =
- vabal_u8(vec_sum_ref1_lo, vget_low_u8(vec_src), vget_low_u8(vec_ref1));
- vec_sum_ref1_hi = vabal_u8(vec_sum_ref1_hi, vget_high_u8(vec_src),
- vget_high_u8(vec_ref1));
- vec_sum_ref2_lo =
- vabal_u8(vec_sum_ref2_lo, vget_low_u8(vec_src), vget_low_u8(vec_ref2));
- vec_sum_ref2_hi = vabal_u8(vec_sum_ref2_hi, vget_high_u8(vec_src),
- vget_high_u8(vec_ref2));
- vec_sum_ref3_lo =
- vabal_u8(vec_sum_ref3_lo, vget_low_u8(vec_src), vget_low_u8(vec_ref3));
- vec_sum_ref3_hi = vabal_u8(vec_sum_ref3_hi, vget_high_u8(vec_src),
- vget_high_u8(vec_ref3));
-
- src += src_stride;
- ref0 += ref_stride;
- ref1 += ref_stride;
- ref2 += ref_stride;
- ref3 += ref_stride;
- }
-
- res[0] = horizontal_long_add_u16x8(vec_sum_ref0_lo, vec_sum_ref0_hi);
- res[1] = horizontal_long_add_u16x8(vec_sum_ref1_lo, vec_sum_ref1_hi);
- res[2] = horizontal_long_add_u16x8(vec_sum_ref2_lo, vec_sum_ref2_hi);
- res[3] = horizontal_long_add_u16x8(vec_sum_ref3_lo, vec_sum_ref3_hi);
-}
-
-static void sad_row4_neon(uint16x4_t *vec_src, const uint8x8_t q0,
- const uint8x8_t ref) {
- uint8x8_t q2 = vabd_u8(q0, ref);
- *vec_src = vpadal_u8(*vec_src, q2);
-}
-
-static void sad_row8_neon(uint16x4_t *vec_src, const uint8x8_t *q0,
- const uint8_t *ref_ptr) {
- uint8x8_t q1 = vld1_u8(ref_ptr);
- uint8x8_t q2 = vabd_u8(*q0, q1);
- *vec_src = vpadal_u8(*vec_src, q2);
-}
-
-static void sad_row16_neon(uint16x8_t *vec_src, const uint8x16_t *q0,
- const uint8_t *ref_ptr) {
- uint8x16_t q1 = vld1q_u8(ref_ptr);
- uint8x16_t q2 = vabdq_u8(*q0, q1);
- *vec_src = vpadalq_u8(*vec_src, q2);
-}
-
-void aom_sadMxNx4d_neon(int width, int height, const uint8_t *src,
- int src_stride, const uint8_t *const ref[4],
- int ref_stride, uint32_t res[4]) {
- const uint8_t *ref0, *ref1, *ref2, *ref3;
-
- ref0 = ref[0];
- ref1 = ref[1];
- ref2 = ref[2];
- ref3 = ref[3];
-
- res[0] = 0;
- res[1] = 0;
- res[2] = 0;
- res[3] = 0;
-
- switch (width) {
- case 4: {
- uint32_t src4, ref40, ref41, ref42, ref43;
- uint32x2_t q8 = vdup_n_u32(0);
- uint32x2_t q4 = vdup_n_u32(0);
- uint32x2_t q5 = vdup_n_u32(0);
- uint32x2_t q6 = vdup_n_u32(0);
- uint32x2_t q7 = vdup_n_u32(0);
-
- for (int i = 0; i < height / 2; i++) {
- uint16x4_t q0 = vdup_n_u16(0);
- uint16x4_t q1 = vdup_n_u16(0);
- uint16x4_t q2 = vdup_n_u16(0);
- uint16x4_t q3 = vdup_n_u16(0);
-
- memcpy(&src4, src, 4);
- memcpy(&ref40, ref0, 4);
- memcpy(&ref41, ref1, 4);
- memcpy(&ref42, ref2, 4);
- memcpy(&ref43, ref3, 4);
-
- src += src_stride;
- ref0 += ref_stride;
- ref1 += ref_stride;
- ref2 += ref_stride;
- ref3 += ref_stride;
-
- q8 = vset_lane_u32(src4, q8, 0);
- q4 = vset_lane_u32(ref40, q4, 0);
- q5 = vset_lane_u32(ref41, q5, 0);
- q6 = vset_lane_u32(ref42, q6, 0);
- q7 = vset_lane_u32(ref43, q7, 0);
-
- memcpy(&src4, src, 4);
- memcpy(&ref40, ref0, 4);
- memcpy(&ref41, ref1, 4);
- memcpy(&ref42, ref2, 4);
- memcpy(&ref43, ref3, 4);
-
- src += src_stride;
- ref0 += ref_stride;
- ref1 += ref_stride;
- ref2 += ref_stride;
- ref3 += ref_stride;
-
- q8 = vset_lane_u32(src4, q8, 1);
- q4 = vset_lane_u32(ref40, q4, 1);
- q5 = vset_lane_u32(ref41, q5, 1);
- q6 = vset_lane_u32(ref42, q6, 1);
- q7 = vset_lane_u32(ref43, q7, 1);
-
- sad_row4_neon(&q0, vreinterpret_u8_u32(q8), vreinterpret_u8_u32(q4));
- sad_row4_neon(&q1, vreinterpret_u8_u32(q8), vreinterpret_u8_u32(q5));
- sad_row4_neon(&q2, vreinterpret_u8_u32(q8), vreinterpret_u8_u32(q6));
- sad_row4_neon(&q3, vreinterpret_u8_u32(q8), vreinterpret_u8_u32(q7));
-
- res[0] += horizontal_add_u16x4(q0);
- res[1] += horizontal_add_u16x4(q1);
- res[2] += horizontal_add_u16x4(q2);
- res[3] += horizontal_add_u16x4(q3);
- }
- break;
- }
- case 8: {
- for (int i = 0; i < height; i++) {
- uint16x4_t q0 = vdup_n_u16(0);
- uint16x4_t q1 = vdup_n_u16(0);
- uint16x4_t q2 = vdup_n_u16(0);
- uint16x4_t q3 = vdup_n_u16(0);
-
- uint8x8_t q5 = vld1_u8(src);
-
- sad_row8_neon(&q0, &q5, ref0);
- sad_row8_neon(&q1, &q5, ref1);
- sad_row8_neon(&q2, &q5, ref2);
- sad_row8_neon(&q3, &q5, ref3);
-
- src += src_stride;
- ref0 += ref_stride;
- ref1 += ref_stride;
- ref2 += ref_stride;
- ref3 += ref_stride;
-
- res[0] += horizontal_add_u16x4(q0);
- res[1] += horizontal_add_u16x4(q1);
- res[2] += horizontal_add_u16x4(q2);
- res[3] += horizontal_add_u16x4(q3);
- }
- break;
- }
- case 16: {
- for (int i = 0; i < height; i++) {
- uint16x8_t q0 = vdupq_n_u16(0);
- uint16x8_t q1 = vdupq_n_u16(0);
- uint16x8_t q2 = vdupq_n_u16(0);
- uint16x8_t q3 = vdupq_n_u16(0);
-
- uint8x16_t q4 = vld1q_u8(src);
-
- sad_row16_neon(&q0, &q4, ref0);
- sad_row16_neon(&q1, &q4, ref1);
- sad_row16_neon(&q2, &q4, ref2);
- sad_row16_neon(&q3, &q4, ref3);
-
- src += src_stride;
- ref0 += ref_stride;
- ref1 += ref_stride;
- ref2 += ref_stride;
- ref3 += ref_stride;
-
- res[0] += horizontal_add_u16x8(q0);
- res[1] += horizontal_add_u16x8(q1);
- res[2] += horizontal_add_u16x8(q2);
- res[3] += horizontal_add_u16x8(q3);
- }
- break;
- }
- case 32: {
- for (int i = 0; i < height; i++) {
- uint16x8_t q0 = vdupq_n_u16(0);
- uint16x8_t q1 = vdupq_n_u16(0);
- uint16x8_t q2 = vdupq_n_u16(0);
- uint16x8_t q3 = vdupq_n_u16(0);
-
- uint8x16_t q4 = vld1q_u8(src);
-
- sad_row16_neon(&q0, &q4, ref0);
- sad_row16_neon(&q1, &q4, ref1);
- sad_row16_neon(&q2, &q4, ref2);
- sad_row16_neon(&q3, &q4, ref3);
-
- q4 = vld1q_u8(src + 16);
-
- sad_row16_neon(&q0, &q4, ref0 + 16);
- sad_row16_neon(&q1, &q4, ref1 + 16);
- sad_row16_neon(&q2, &q4, ref2 + 16);
- sad_row16_neon(&q3, &q4, ref3 + 16);
-
- src += src_stride;
- ref0 += ref_stride;
- ref1 += ref_stride;
- ref2 += ref_stride;
- ref3 += ref_stride;
-
- res[0] += horizontal_add_u16x8(q0);
- res[1] += horizontal_add_u16x8(q1);
- res[2] += horizontal_add_u16x8(q2);
- res[3] += horizontal_add_u16x8(q3);
- }
- break;
- }
- case 64: {
- for (int i = 0; i < height; i++) {
- uint16x8_t q0 = vdupq_n_u16(0);
- uint16x8_t q1 = vdupq_n_u16(0);
- uint16x8_t q2 = vdupq_n_u16(0);
- uint16x8_t q3 = vdupq_n_u16(0);
-
- uint8x16_t q4 = vld1q_u8(src);
-
- sad_row16_neon(&q0, &q4, ref0);
- sad_row16_neon(&q1, &q4, ref1);
- sad_row16_neon(&q2, &q4, ref2);
- sad_row16_neon(&q3, &q4, ref3);
-
- q4 = vld1q_u8(src + 16);
-
- sad_row16_neon(&q0, &q4, ref0 + 16);
- sad_row16_neon(&q1, &q4, ref1 + 16);
- sad_row16_neon(&q2, &q4, ref2 + 16);
- sad_row16_neon(&q3, &q4, ref3 + 16);
-
- q4 = vld1q_u8(src + 32);
-
- sad_row16_neon(&q0, &q4, ref0 + 32);
- sad_row16_neon(&q1, &q4, ref1 + 32);
- sad_row16_neon(&q2, &q4, ref2 + 32);
- sad_row16_neon(&q3, &q4, ref3 + 32);
-
- q4 = vld1q_u8(src + 48);
-
- sad_row16_neon(&q0, &q4, ref0 + 48);
- sad_row16_neon(&q1, &q4, ref1 + 48);
- sad_row16_neon(&q2, &q4, ref2 + 48);
- sad_row16_neon(&q3, &q4, ref3 + 48);
-
- src += src_stride;
- ref0 += ref_stride;
- ref1 += ref_stride;
- ref2 += ref_stride;
- ref3 += ref_stride;
-
- res[0] += horizontal_add_u16x8(q0);
- res[1] += horizontal_add_u16x8(q1);
- res[2] += horizontal_add_u16x8(q2);
- res[3] += horizontal_add_u16x8(q3);
- }
- break;
- }
- case 128: {
- for (int i = 0; i < height; i++) {
- uint16x8_t q0 = vdupq_n_u16(0);
- uint16x8_t q1 = vdupq_n_u16(0);
- uint16x8_t q2 = vdupq_n_u16(0);
- uint16x8_t q3 = vdupq_n_u16(0);
-
- uint8x16_t q4 = vld1q_u8(src);
-
- sad_row16_neon(&q0, &q4, ref0);
- sad_row16_neon(&q1, &q4, ref1);
- sad_row16_neon(&q2, &q4, ref2);
- sad_row16_neon(&q3, &q4, ref3);
-
- q4 = vld1q_u8(src + 16);
-
- sad_row16_neon(&q0, &q4, ref0 + 16);
- sad_row16_neon(&q1, &q4, ref1 + 16);
- sad_row16_neon(&q2, &q4, ref2 + 16);
- sad_row16_neon(&q3, &q4, ref3 + 16);
-
- q4 = vld1q_u8(src + 32);
-
- sad_row16_neon(&q0, &q4, ref0 + 32);
- sad_row16_neon(&q1, &q4, ref1 + 32);
- sad_row16_neon(&q2, &q4, ref2 + 32);
- sad_row16_neon(&q3, &q4, ref3 + 32);
-
- q4 = vld1q_u8(src + 48);
-
- sad_row16_neon(&q0, &q4, ref0 + 48);
- sad_row16_neon(&q1, &q4, ref1 + 48);
- sad_row16_neon(&q2, &q4, ref2 + 48);
- sad_row16_neon(&q3, &q4, ref3 + 48);
-
- q4 = vld1q_u8(src + 64);
-
- sad_row16_neon(&q0, &q4, ref0 + 64);
- sad_row16_neon(&q1, &q4, ref1 + 64);
- sad_row16_neon(&q2, &q4, ref2 + 64);
- sad_row16_neon(&q3, &q4, ref3 + 64);
-
- q4 = vld1q_u8(src + 80);
-
- sad_row16_neon(&q0, &q4, ref0 + 80);
- sad_row16_neon(&q1, &q4, ref1 + 80);
- sad_row16_neon(&q2, &q4, ref2 + 80);
- sad_row16_neon(&q3, &q4, ref3 + 80);
-
- q4 = vld1q_u8(src + 96);
-
- sad_row16_neon(&q0, &q4, ref0 + 96);
- sad_row16_neon(&q1, &q4, ref1 + 96);
- sad_row16_neon(&q2, &q4, ref2 + 96);
- sad_row16_neon(&q3, &q4, ref3 + 96);
-
- q4 = vld1q_u8(src + 112);
-
- sad_row16_neon(&q0, &q4, ref0 + 112);
- sad_row16_neon(&q1, &q4, ref1 + 112);
- sad_row16_neon(&q2, &q4, ref2 + 112);
- sad_row16_neon(&q3, &q4, ref3 + 112);
-
- src += src_stride;
- ref0 += ref_stride;
- ref1 += ref_stride;
- ref2 += ref_stride;
- ref3 += ref_stride;
-
- res[0] += horizontal_add_u16x8(q0);
- res[1] += horizontal_add_u16x8(q1);
- res[2] += horizontal_add_u16x8(q2);
- res[3] += horizontal_add_u16x8(q3);
- }
- }
- }
-}
-
-#define SAD_SKIP_MXN_NEON(m, n) \
- void aom_sad_skip_##m##x##n##x4d_neon(const uint8_t *src, int src_stride, \
+#define SAD_SKIP_WXH_4D_NEON(w, h) \
+ void aom_sad_skip_##w##x##h##x4d_neon(const uint8_t *src, int src_stride, \
const uint8_t *const ref[4], \
int ref_stride, uint32_t res[4]) { \
- aom_sadMxNx4d_neon(m, ((n) >> 1), src, 2 * src_stride, ref, \
- 2 * ref_stride, res); \
+ sad##w##xhx4d_neon(src, 2 * src_stride, ref, 2 * ref_stride, res, \
+ ((h) >> 1)); \
res[0] <<= 1; \
res[1] <<= 1; \
res[2] <<= 1; \
res[3] <<= 1; \
}
-SAD_SKIP_MXN_NEON(4, 8)
-SAD_SKIP_MXN_NEON(4, 16)
-SAD_SKIP_MXN_NEON(4, 32)
+SAD_SKIP_WXH_4D_NEON(4, 8)
+SAD_SKIP_WXH_4D_NEON(4, 16)
+SAD_SKIP_WXH_4D_NEON(4, 32)
-SAD_SKIP_MXN_NEON(8, 8)
-SAD_SKIP_MXN_NEON(8, 16)
-SAD_SKIP_MXN_NEON(8, 32)
+SAD_SKIP_WXH_4D_NEON(8, 8)
+SAD_SKIP_WXH_4D_NEON(8, 16)
+SAD_SKIP_WXH_4D_NEON(8, 32)
-SAD_SKIP_MXN_NEON(16, 8)
-SAD_SKIP_MXN_NEON(16, 16)
-SAD_SKIP_MXN_NEON(16, 32)
-SAD_SKIP_MXN_NEON(16, 64)
+SAD_SKIP_WXH_4D_NEON(16, 8)
+SAD_SKIP_WXH_4D_NEON(16, 16)
+SAD_SKIP_WXH_4D_NEON(16, 32)
+SAD_SKIP_WXH_4D_NEON(16, 64)
-SAD_SKIP_MXN_NEON(32, 8)
-SAD_SKIP_MXN_NEON(32, 16)
-SAD_SKIP_MXN_NEON(32, 32)
-SAD_SKIP_MXN_NEON(32, 64)
+SAD_SKIP_WXH_4D_NEON(32, 8)
+SAD_SKIP_WXH_4D_NEON(32, 16)
+SAD_SKIP_WXH_4D_NEON(32, 32)
+SAD_SKIP_WXH_4D_NEON(32, 64)
-SAD_SKIP_MXN_NEON(64, 16)
-SAD_SKIP_MXN_NEON(64, 32)
-SAD_SKIP_MXN_NEON(64, 64)
-SAD_SKIP_MXN_NEON(64, 128)
+SAD_SKIP_WXH_4D_NEON(64, 16)
+SAD_SKIP_WXH_4D_NEON(64, 32)
+SAD_SKIP_WXH_4D_NEON(64, 64)
+SAD_SKIP_WXH_4D_NEON(64, 128)
-SAD_SKIP_MXN_NEON(128, 64)
-SAD_SKIP_MXN_NEON(128, 128)
+SAD_SKIP_WXH_4D_NEON(128, 64)
+SAD_SKIP_WXH_4D_NEON(128, 128)
-#undef SAD_SKIP_MXN_NEON
+#undef SAD_SKIP_WXH_4D_NEON
diff --git a/aom_dsp/arm/sad_neon.c b/aom_dsp/arm/sad_neon.c
index acd2c54..47e0045 100644
--- a/aom_dsp/arm/sad_neon.c
+++ b/aom_dsp/arm/sad_neon.c
@@ -15,531 +15,378 @@
#include "aom/aom_integer.h"
#include "aom_dsp/arm/sum_neon.h"
-unsigned int aom_sad8x16_neon(const uint8_t *src_ptr, int src_stride,
- const uint8_t *ref_ptr, int ref_stride) {
- uint8x8_t d0, d8;
- uint16x8_t q12;
- uint32x4_t q1;
- uint64x2_t q3;
- uint32x2_t d5;
- int i;
+#if defined(__ARM_FEATURE_DOTPROD)
- d0 = vld1_u8(src_ptr);
- src_ptr += src_stride;
- d8 = vld1_u8(ref_ptr);
- ref_ptr += ref_stride;
- q12 = vabdl_u8(d0, d8);
+static INLINE unsigned int sadwxh_neon(const uint8_t *src_ptr, int src_stride,
+ const uint8_t *ref_ptr, int ref_stride,
+ int w, int h) {
+ // Only two accumulators are required for optimal instruction throughput of
+ // the ABD, UDOT sequence on CPUs with either 2 or 4 Neon pipes.
+ uint32x4_t sum[2] = { vdupq_n_u32(0), vdupq_n_u32(0) };
- for (i = 0; i < 15; i++) {
- d0 = vld1_u8(src_ptr);
+ int i = 0;
+ do {
+ int j = 0;
+ do {
+ uint8x16_t s0, s1, r0, r1, diff0, diff1;
+
+ s0 = vld1q_u8(src_ptr + j);
+ r0 = vld1q_u8(ref_ptr + j);
+ diff0 = vabdq_u8(s0, r0);
+ sum[0] = vdotq_u32(sum[0], diff0, vdupq_n_u8(1));
+
+ s1 = vld1q_u8(src_ptr + j + 16);
+ r1 = vld1q_u8(ref_ptr + j + 16);
+ diff1 = vabdq_u8(s1, r1);
+ sum[1] = vdotq_u32(sum[1], diff1, vdupq_n_u8(1));
+
+ j += 32;
+ } while (j < w);
+
src_ptr += src_stride;
- d8 = vld1_u8(ref_ptr);
ref_ptr += ref_stride;
- q12 = vabal_u8(q12, d0, d8);
- }
+ i++;
+ } while (i < h);
- q1 = vpaddlq_u16(q12);
- q3 = vpaddlq_u32(q1);
- d5 = vadd_u32(vreinterpret_u32_u64(vget_low_u64(q3)),
- vreinterpret_u32_u64(vget_high_u64(q3)));
-
- return vget_lane_u32(d5, 0);
-}
-
-unsigned int aom_sad4x4_neon(const uint8_t *src_ptr, int src_stride,
- const uint8_t *ref_ptr, int ref_stride) {
- uint8x8_t d0, d8;
- uint16x8_t q12;
- uint32x2_t d1;
- uint64x1_t d3;
- int i;
-
- d0 = vld1_u8(src_ptr);
- src_ptr += src_stride;
- d8 = vld1_u8(ref_ptr);
- ref_ptr += ref_stride;
- q12 = vabdl_u8(d0, d8);
-
- for (i = 0; i < 3; i++) {
- d0 = vld1_u8(src_ptr);
- src_ptr += src_stride;
- d8 = vld1_u8(ref_ptr);
- ref_ptr += ref_stride;
- q12 = vabal_u8(q12, d0, d8);
- }
-
- d1 = vpaddl_u16(vget_low_u16(q12));
- d3 = vpaddl_u32(d1);
-
- return vget_lane_u32(vreinterpret_u32_u64(d3), 0);
-}
-
-unsigned int aom_sad16x8_neon(const uint8_t *src_ptr, int src_stride,
- const uint8_t *ref_ptr, int ref_stride) {
- uint8x16_t q0, q4;
- uint16x8_t q12, q13;
- uint32x4_t q1;
- uint64x2_t q3;
- uint32x2_t d5;
- int i;
-
- q0 = vld1q_u8(src_ptr);
- src_ptr += src_stride;
- q4 = vld1q_u8(ref_ptr);
- ref_ptr += ref_stride;
- q12 = vabdl_u8(vget_low_u8(q0), vget_low_u8(q4));
- q13 = vabdl_u8(vget_high_u8(q0), vget_high_u8(q4));
-
- for (i = 0; i < 7; i++) {
- q0 = vld1q_u8(src_ptr);
- src_ptr += src_stride;
- q4 = vld1q_u8(ref_ptr);
- ref_ptr += ref_stride;
- q12 = vabal_u8(q12, vget_low_u8(q0), vget_low_u8(q4));
- q13 = vabal_u8(q13, vget_high_u8(q0), vget_high_u8(q4));
- }
-
- q12 = vaddq_u16(q12, q13);
- q1 = vpaddlq_u16(q12);
- q3 = vpaddlq_u32(q1);
- d5 = vadd_u32(vreinterpret_u32_u64(vget_low_u64(q3)),
- vreinterpret_u32_u64(vget_high_u64(q3)));
-
- return vget_lane_u32(d5, 0);
-}
-
-unsigned int aom_sad64x64_neon(const uint8_t *src, int src_stride,
- const uint8_t *ref, int ref_stride) {
- int i;
- uint16x8_t vec_accum_lo = vdupq_n_u16(0);
- uint16x8_t vec_accum_hi = vdupq_n_u16(0);
- for (i = 0; i < 64; ++i) {
- const uint8x16_t vec_src_00 = vld1q_u8(src);
- const uint8x16_t vec_src_16 = vld1q_u8(src + 16);
- const uint8x16_t vec_src_32 = vld1q_u8(src + 32);
- const uint8x16_t vec_src_48 = vld1q_u8(src + 48);
- const uint8x16_t vec_ref_00 = vld1q_u8(ref);
- const uint8x16_t vec_ref_16 = vld1q_u8(ref + 16);
- const uint8x16_t vec_ref_32 = vld1q_u8(ref + 32);
- const uint8x16_t vec_ref_48 = vld1q_u8(ref + 48);
- src += src_stride;
- ref += ref_stride;
- vec_accum_lo = vabal_u8(vec_accum_lo, vget_low_u8(vec_src_00),
- vget_low_u8(vec_ref_00));
- vec_accum_hi = vabal_u8(vec_accum_hi, vget_high_u8(vec_src_00),
- vget_high_u8(vec_ref_00));
- vec_accum_lo = vabal_u8(vec_accum_lo, vget_low_u8(vec_src_16),
- vget_low_u8(vec_ref_16));
- vec_accum_hi = vabal_u8(vec_accum_hi, vget_high_u8(vec_src_16),
- vget_high_u8(vec_ref_16));
- vec_accum_lo = vabal_u8(vec_accum_lo, vget_low_u8(vec_src_32),
- vget_low_u8(vec_ref_32));
- vec_accum_hi = vabal_u8(vec_accum_hi, vget_high_u8(vec_src_32),
- vget_high_u8(vec_ref_32));
- vec_accum_lo = vabal_u8(vec_accum_lo, vget_low_u8(vec_src_48),
- vget_low_u8(vec_ref_48));
- vec_accum_hi = vabal_u8(vec_accum_hi, vget_high_u8(vec_src_48),
- vget_high_u8(vec_ref_48));
- }
- return horizontal_long_add_u16x8(vec_accum_lo, vec_accum_hi);
-}
-
-unsigned int aom_sad128x128_neon(const uint8_t *src, int src_stride,
- const uint8_t *ref, int ref_stride) {
- uint16x8_t vec_accum_lo, vec_accum_hi;
- uint32x4_t vec_accum_32lo = vdupq_n_u32(0);
- uint32x4_t vec_accum_32hi = vdupq_n_u32(0);
- uint16x8_t tmp;
- for (int i = 0; i < 128; ++i) {
- const uint8x16_t vec_src_00 = vld1q_u8(src);
- const uint8x16_t vec_src_16 = vld1q_u8(src + 16);
- const uint8x16_t vec_src_32 = vld1q_u8(src + 32);
- const uint8x16_t vec_src_48 = vld1q_u8(src + 48);
- const uint8x16_t vec_src_64 = vld1q_u8(src + 64);
- const uint8x16_t vec_src_80 = vld1q_u8(src + 80);
- const uint8x16_t vec_src_96 = vld1q_u8(src + 96);
- const uint8x16_t vec_src_112 = vld1q_u8(src + 112);
- const uint8x16_t vec_ref_00 = vld1q_u8(ref);
- const uint8x16_t vec_ref_16 = vld1q_u8(ref + 16);
- const uint8x16_t vec_ref_32 = vld1q_u8(ref + 32);
- const uint8x16_t vec_ref_48 = vld1q_u8(ref + 48);
- const uint8x16_t vec_ref_64 = vld1q_u8(ref + 64);
- const uint8x16_t vec_ref_80 = vld1q_u8(ref + 80);
- const uint8x16_t vec_ref_96 = vld1q_u8(ref + 96);
- const uint8x16_t vec_ref_112 = vld1q_u8(ref + 112);
- src += src_stride;
- ref += ref_stride;
- vec_accum_lo = vdupq_n_u16(0);
- vec_accum_hi = vdupq_n_u16(0);
- vec_accum_lo = vabal_u8(vec_accum_lo, vget_low_u8(vec_src_00),
- vget_low_u8(vec_ref_00));
- vec_accum_hi = vabal_u8(vec_accum_hi, vget_high_u8(vec_src_00),
- vget_high_u8(vec_ref_00));
- vec_accum_lo = vabal_u8(vec_accum_lo, vget_low_u8(vec_src_16),
- vget_low_u8(vec_ref_16));
- vec_accum_hi = vabal_u8(vec_accum_hi, vget_high_u8(vec_src_16),
- vget_high_u8(vec_ref_16));
- vec_accum_lo = vabal_u8(vec_accum_lo, vget_low_u8(vec_src_32),
- vget_low_u8(vec_ref_32));
- vec_accum_hi = vabal_u8(vec_accum_hi, vget_high_u8(vec_src_32),
- vget_high_u8(vec_ref_32));
- vec_accum_lo = vabal_u8(vec_accum_lo, vget_low_u8(vec_src_48),
- vget_low_u8(vec_ref_48));
- vec_accum_hi = vabal_u8(vec_accum_hi, vget_high_u8(vec_src_48),
- vget_high_u8(vec_ref_48));
- vec_accum_lo = vabal_u8(vec_accum_lo, vget_low_u8(vec_src_64),
- vget_low_u8(vec_ref_64));
- vec_accum_hi = vabal_u8(vec_accum_hi, vget_high_u8(vec_src_64),
- vget_high_u8(vec_ref_64));
- vec_accum_lo = vabal_u8(vec_accum_lo, vget_low_u8(vec_src_80),
- vget_low_u8(vec_ref_80));
- vec_accum_hi = vabal_u8(vec_accum_hi, vget_high_u8(vec_src_80),
- vget_high_u8(vec_ref_80));
- vec_accum_lo = vabal_u8(vec_accum_lo, vget_low_u8(vec_src_96),
- vget_low_u8(vec_ref_96));
- vec_accum_hi = vabal_u8(vec_accum_hi, vget_high_u8(vec_src_96),
- vget_high_u8(vec_ref_96));
- vec_accum_lo = vabal_u8(vec_accum_lo, vget_low_u8(vec_src_112),
- vget_low_u8(vec_ref_112));
- vec_accum_hi = vabal_u8(vec_accum_hi, vget_high_u8(vec_src_112),
- vget_high_u8(vec_ref_112));
-
- tmp = vaddq_u16(vec_accum_lo, vec_accum_hi);
- vec_accum_32lo = vaddw_u16(vec_accum_32lo, vget_low_u16(tmp));
- vec_accum_32hi = vaddw_u16(vec_accum_32hi, vget_high_u16(tmp));
- }
- const uint32x4_t a = vaddq_u32(vec_accum_32lo, vec_accum_32hi);
- const uint64x2_t b = vpaddlq_u32(a);
- const uint32x2_t c = vadd_u32(vreinterpret_u32_u64(vget_low_u64(b)),
- vreinterpret_u32_u64(vget_high_u64(b)));
- return vget_lane_u32(c, 0);
-}
-
-unsigned int aom_sad32x32_neon(const uint8_t *src, int src_stride,
- const uint8_t *ref, int ref_stride) {
- int i;
- uint16x8_t vec_accum_lo = vdupq_n_u16(0);
- uint16x8_t vec_accum_hi = vdupq_n_u16(0);
-
- for (i = 0; i < 32; ++i) {
- const uint8x16_t vec_src_00 = vld1q_u8(src);
- const uint8x16_t vec_src_16 = vld1q_u8(src + 16);
- const uint8x16_t vec_ref_00 = vld1q_u8(ref);
- const uint8x16_t vec_ref_16 = vld1q_u8(ref + 16);
- src += src_stride;
- ref += ref_stride;
- vec_accum_lo = vabal_u8(vec_accum_lo, vget_low_u8(vec_src_00),
- vget_low_u8(vec_ref_00));
- vec_accum_hi = vabal_u8(vec_accum_hi, vget_high_u8(vec_src_00),
- vget_high_u8(vec_ref_00));
- vec_accum_lo = vabal_u8(vec_accum_lo, vget_low_u8(vec_src_16),
- vget_low_u8(vec_ref_16));
- vec_accum_hi = vabal_u8(vec_accum_hi, vget_high_u8(vec_src_16),
- vget_high_u8(vec_ref_16));
- }
- return horizontal_add_u16x8(vaddq_u16(vec_accum_lo, vec_accum_hi));
-}
-
-unsigned int aom_sad16x16_neon(const uint8_t *src, int src_stride,
- const uint8_t *ref, int ref_stride) {
- int i;
- uint16x8_t vec_accum_lo = vdupq_n_u16(0);
- uint16x8_t vec_accum_hi = vdupq_n_u16(0);
-
- for (i = 0; i < 16; ++i) {
- const uint8x16_t vec_src = vld1q_u8(src);
- const uint8x16_t vec_ref = vld1q_u8(ref);
- src += src_stride;
- ref += ref_stride;
- vec_accum_lo =
- vabal_u8(vec_accum_lo, vget_low_u8(vec_src), vget_low_u8(vec_ref));
- vec_accum_hi =
- vabal_u8(vec_accum_hi, vget_high_u8(vec_src), vget_high_u8(vec_ref));
- }
- return horizontal_add_u16x8(vaddq_u16(vec_accum_lo, vec_accum_hi));
-}
-
-unsigned int aom_sad8x8_neon(const uint8_t *src, int src_stride,
- const uint8_t *ref, int ref_stride) {
- int i;
- uint16x8_t vec_accum = vdupq_n_u16(0);
-
- for (i = 0; i < 8; ++i) {
- const uint8x8_t vec_src = vld1_u8(src);
- const uint8x8_t vec_ref = vld1_u8(ref);
- src += src_stride;
- ref += ref_stride;
- vec_accum = vabal_u8(vec_accum, vec_src, vec_ref);
- }
- return horizontal_add_u16x8(vec_accum);
+ return horizontal_add_u32x4(vaddq_u32(sum[0], sum[1]));
}
static INLINE unsigned int sad128xh_neon(const uint8_t *src_ptr, int src_stride,
const uint8_t *ref_ptr, int ref_stride,
int h) {
- int sum = 0;
- for (int i = 0; i < h; i++) {
- uint16x8_t q3 = vdupq_n_u16(0);
-
- uint8x16_t q0 = vld1q_u8(src_ptr);
- uint8x16_t q1 = vld1q_u8(ref_ptr);
- uint8x16_t q2 = vabdq_u8(q0, q1);
- q3 = vpadalq_u8(q3, q2);
-
- q0 = vld1q_u8(src_ptr + 16);
- q1 = vld1q_u8(ref_ptr + 16);
- q2 = vabdq_u8(q0, q1);
- q3 = vpadalq_u8(q3, q2);
-
- q0 = vld1q_u8(src_ptr + 32);
- q1 = vld1q_u8(ref_ptr + 32);
- q2 = vabdq_u8(q0, q1);
- q3 = vpadalq_u8(q3, q2);
-
- q0 = vld1q_u8(src_ptr + 48);
- q1 = vld1q_u8(ref_ptr + 48);
- q2 = vabdq_u8(q0, q1);
- q3 = vpadalq_u8(q3, q2);
-
- q0 = vld1q_u8(src_ptr + 64);
- q1 = vld1q_u8(ref_ptr + 64);
- q2 = vabdq_u8(q0, q1);
- q3 = vpadalq_u8(q3, q2);
-
- q0 = vld1q_u8(src_ptr + 80);
- q1 = vld1q_u8(ref_ptr + 80);
- q2 = vabdq_u8(q0, q1);
- q3 = vpadalq_u8(q3, q2);
-
- q0 = vld1q_u8(src_ptr + 96);
- q1 = vld1q_u8(ref_ptr + 96);
- q2 = vabdq_u8(q0, q1);
- q3 = vpadalq_u8(q3, q2);
-
- q0 = vld1q_u8(src_ptr + 112);
- q1 = vld1q_u8(ref_ptr + 112);
- q2 = vabdq_u8(q0, q1);
- q3 = vpadalq_u8(q3, q2);
-
- src_ptr += src_stride;
- ref_ptr += ref_stride;
-
- sum += horizontal_add_u16x8(q3);
- }
-
- return sum;
+ return sadwxh_neon(src_ptr, src_stride, ref_ptr, ref_stride, 128, h);
}
static INLINE unsigned int sad64xh_neon(const uint8_t *src_ptr, int src_stride,
const uint8_t *ref_ptr, int ref_stride,
int h) {
- int sum = 0;
- for (int i = 0; i < h; i++) {
- uint16x8_t q3 = vdupq_n_u16(0);
-
- uint8x16_t q0 = vld1q_u8(src_ptr);
- uint8x16_t q1 = vld1q_u8(ref_ptr);
- uint8x16_t q2 = vabdq_u8(q0, q1);
- q3 = vpadalq_u8(q3, q2);
-
- q0 = vld1q_u8(src_ptr + 16);
- q1 = vld1q_u8(ref_ptr + 16);
- q2 = vabdq_u8(q0, q1);
- q3 = vpadalq_u8(q3, q2);
-
- q0 = vld1q_u8(src_ptr + 32);
- q1 = vld1q_u8(ref_ptr + 32);
- q2 = vabdq_u8(q0, q1);
- q3 = vpadalq_u8(q3, q2);
-
- q0 = vld1q_u8(src_ptr + 48);
- q1 = vld1q_u8(ref_ptr + 48);
- q2 = vabdq_u8(q0, q1);
- q3 = vpadalq_u8(q3, q2);
-
- src_ptr += src_stride;
- ref_ptr += ref_stride;
-
- sum += horizontal_add_u16x8(q3);
- }
-
- return sum;
+ return sadwxh_neon(src_ptr, src_stride, ref_ptr, ref_stride, 64, h);
}
static INLINE unsigned int sad32xh_neon(const uint8_t *src_ptr, int src_stride,
const uint8_t *ref_ptr, int ref_stride,
int h) {
- int sum = 0;
- for (int i = 0; i < h; i++) {
- uint16x8_t q3 = vdupq_n_u16(0);
-
- uint8x16_t q0 = vld1q_u8(src_ptr);
- uint8x16_t q1 = vld1q_u8(ref_ptr);
- uint8x16_t q2 = vabdq_u8(q0, q1);
- q3 = vpadalq_u8(q3, q2);
-
- q0 = vld1q_u8(src_ptr + 16);
- q1 = vld1q_u8(ref_ptr + 16);
- q2 = vabdq_u8(q0, q1);
- q3 = vpadalq_u8(q3, q2);
-
- sum += horizontal_add_u16x8(q3);
-
- src_ptr += src_stride;
- ref_ptr += ref_stride;
- }
-
- return sum;
+ return sadwxh_neon(src_ptr, src_stride, ref_ptr, ref_stride, 32, h);
}
static INLINE unsigned int sad16xh_neon(const uint8_t *src_ptr, int src_stride,
const uint8_t *ref_ptr, int ref_stride,
int h) {
- int sum = 0;
- for (int i = 0; i < h; i++) {
- uint8x8_t q0 = vld1_u8(src_ptr);
- uint8x8_t q1 = vld1_u8(ref_ptr);
- sum += vget_lane_u16(vpaddl_u8(vabd_u8(q0, q1)), 0);
- sum += vget_lane_u16(vpaddl_u8(vabd_u8(q0, q1)), 1);
- sum += vget_lane_u16(vpaddl_u8(vabd_u8(q0, q1)), 2);
- sum += vget_lane_u16(vpaddl_u8(vabd_u8(q0, q1)), 3);
- q0 = vld1_u8(src_ptr + 8);
- q1 = vld1_u8(ref_ptr + 8);
- sum += vget_lane_u16(vpaddl_u8(vabd_u8(q0, q1)), 0);
- sum += vget_lane_u16(vpaddl_u8(vabd_u8(q0, q1)), 1);
- sum += vget_lane_u16(vpaddl_u8(vabd_u8(q0, q1)), 2);
- sum += vget_lane_u16(vpaddl_u8(vabd_u8(q0, q1)), 3);
+ uint32x4_t sum[2] = { vdupq_n_u32(0), vdupq_n_u32(0) };
+
+ int i = 0;
+ do {
+ uint8x16_t s0, s1, r0, r1, diff0, diff1;
+
+ s0 = vld1q_u8(src_ptr);
+ r0 = vld1q_u8(ref_ptr);
+ diff0 = vabdq_u8(s0, r0);
+ sum[0] = vdotq_u32(sum[0], diff0, vdupq_n_u8(1));
src_ptr += src_stride;
ref_ptr += ref_stride;
- }
- return sum;
+ s1 = vld1q_u8(src_ptr);
+ r1 = vld1q_u8(ref_ptr);
+ diff1 = vabdq_u8(s1, r1);
+ sum[1] = vdotq_u32(sum[1], diff1, vdupq_n_u8(1));
+
+ src_ptr += src_stride;
+ ref_ptr += ref_stride;
+ i++;
+ } while (i < h / 2);
+
+ return horizontal_add_u32x4(vaddq_u32(sum[0], sum[1]));
}
+#else // !defined(__ARM_FEATURE_DOTPROD)
+
+static INLINE unsigned int sad128xh_neon(const uint8_t *src_ptr, int src_stride,
+ const uint8_t *ref_ptr, int ref_stride,
+ int h) {
+ // We use 8 accumulators to prevent overflow for large values of 'h', as well
+ // as enabling optimal UADALP instruction throughput on CPUs that have either
+ // 2 or 4 Neon pipes.
+ uint16x8_t sum[8] = { vdupq_n_u16(0), vdupq_n_u16(0), vdupq_n_u16(0),
+ vdupq_n_u16(0), vdupq_n_u16(0), vdupq_n_u16(0),
+ vdupq_n_u16(0), vdupq_n_u16(0) };
+
+ int i = 0;
+ do {
+ uint8x16_t s0, s1, s2, s3, s4, s5, s6, s7;
+ uint8x16_t r0, r1, r2, r3, r4, r5, r6, r7;
+ uint8x16_t diff0, diff1, diff2, diff3, diff4, diff5, diff6, diff7;
+
+ s0 = vld1q_u8(src_ptr);
+ r0 = vld1q_u8(ref_ptr);
+ diff0 = vabdq_u8(s0, r0);
+ sum[0] = vpadalq_u8(sum[0], diff0);
+
+ s1 = vld1q_u8(src_ptr + 16);
+ r1 = vld1q_u8(ref_ptr + 16);
+ diff1 = vabdq_u8(s1, r1);
+ sum[1] = vpadalq_u8(sum[1], diff1);
+
+ s2 = vld1q_u8(src_ptr + 32);
+ r2 = vld1q_u8(ref_ptr + 32);
+ diff2 = vabdq_u8(s2, r2);
+ sum[2] = vpadalq_u8(sum[2], diff2);
+
+ s3 = vld1q_u8(src_ptr + 48);
+ r3 = vld1q_u8(ref_ptr + 48);
+ diff3 = vabdq_u8(s3, r3);
+ sum[3] = vpadalq_u8(sum[3], diff3);
+
+ s4 = vld1q_u8(src_ptr + 64);
+ r4 = vld1q_u8(ref_ptr + 64);
+ diff4 = vabdq_u8(s4, r4);
+ sum[4] = vpadalq_u8(sum[4], diff4);
+
+ s5 = vld1q_u8(src_ptr + 80);
+ r5 = vld1q_u8(ref_ptr + 80);
+ diff5 = vabdq_u8(s5, r5);
+ sum[5] = vpadalq_u8(sum[5], diff5);
+
+ s6 = vld1q_u8(src_ptr + 96);
+ r6 = vld1q_u8(ref_ptr + 96);
+ diff6 = vabdq_u8(s6, r6);
+ sum[6] = vpadalq_u8(sum[6], diff6);
+
+ s7 = vld1q_u8(src_ptr + 112);
+ r7 = vld1q_u8(ref_ptr + 112);
+ diff7 = vabdq_u8(s7, r7);
+ sum[7] = vpadalq_u8(sum[7], diff7);
+
+ src_ptr += src_stride;
+ ref_ptr += ref_stride;
+ i++;
+ } while (i < h);
+
+ uint32x4_t sum_u32 = vpaddlq_u16(sum[0]);
+ sum_u32 = vpadalq_u16(sum_u32, sum[1]);
+ sum_u32 = vpadalq_u16(sum_u32, sum[2]);
+ sum_u32 = vpadalq_u16(sum_u32, sum[3]);
+ sum_u32 = vpadalq_u16(sum_u32, sum[4]);
+ sum_u32 = vpadalq_u16(sum_u32, sum[5]);
+ sum_u32 = vpadalq_u16(sum_u32, sum[6]);
+ sum_u32 = vpadalq_u16(sum_u32, sum[7]);
+
+ return horizontal_add_u32x4(sum_u32);
+}
+
+static INLINE unsigned int sad64xh_neon(const uint8_t *src_ptr, int src_stride,
+ const uint8_t *ref_ptr, int ref_stride,
+ int h) {
+ uint16x8_t sum[4] = { vdupq_n_u16(0), vdupq_n_u16(0), vdupq_n_u16(0),
+ vdupq_n_u16(0) };
+
+ int i = 0;
+ do {
+ uint8x16_t s0, s1, s2, s3, r0, r1, r2, r3;
+ uint8x16_t diff0, diff1, diff2, diff3;
+
+ s0 = vld1q_u8(src_ptr);
+ r0 = vld1q_u8(ref_ptr);
+ diff0 = vabdq_u8(s0, r0);
+ sum[0] = vpadalq_u8(sum[0], diff0);
+
+ s1 = vld1q_u8(src_ptr + 16);
+ r1 = vld1q_u8(ref_ptr + 16);
+ diff1 = vabdq_u8(s1, r1);
+ sum[1] = vpadalq_u8(sum[1], diff1);
+
+ s2 = vld1q_u8(src_ptr + 32);
+ r2 = vld1q_u8(ref_ptr + 32);
+ diff2 = vabdq_u8(s2, r2);
+ sum[2] = vpadalq_u8(sum[2], diff2);
+
+ s3 = vld1q_u8(src_ptr + 48);
+ r3 = vld1q_u8(ref_ptr + 48);
+ diff3 = vabdq_u8(s3, r3);
+ sum[3] = vpadalq_u8(sum[3], diff3);
+
+ src_ptr += src_stride;
+ ref_ptr += ref_stride;
+ i++;
+ } while (i < h);
+
+ uint32x4_t sum_u32 = vpaddlq_u16(sum[0]);
+ sum_u32 = vpadalq_u16(sum_u32, sum[1]);
+ sum_u32 = vpadalq_u16(sum_u32, sum[2]);
+ sum_u32 = vpadalq_u16(sum_u32, sum[3]);
+
+ return horizontal_add_u32x4(sum_u32);
+}
+
+static INLINE unsigned int sad32xh_neon(const uint8_t *src_ptr, int src_stride,
+ const uint8_t *ref_ptr, int ref_stride,
+ int h) {
+ uint32x4_t sum = vdupq_n_u32(0);
+
+ int i = 0;
+ do {
+ uint8x16_t s0 = vld1q_u8(src_ptr);
+ uint8x16_t r0 = vld1q_u8(ref_ptr);
+ uint8x16_t diff0 = vabdq_u8(s0, r0);
+ uint16x8_t sum0 = vpaddlq_u8(diff0);
+
+ uint8x16_t s1 = vld1q_u8(src_ptr + 16);
+ uint8x16_t r1 = vld1q_u8(ref_ptr + 16);
+ uint8x16_t diff1 = vabdq_u8(s1, r1);
+ uint16x8_t sum1 = vpaddlq_u8(diff1);
+
+ sum = vpadalq_u16(sum, sum0);
+ sum = vpadalq_u16(sum, sum1);
+
+ src_ptr += src_stride;
+ ref_ptr += ref_stride;
+ i++;
+ } while (i < h);
+
+ return horizontal_add_u32x4(sum);
+}
+
+static INLINE unsigned int sad16xh_neon(const uint8_t *src_ptr, int src_stride,
+ const uint8_t *ref_ptr, int ref_stride,
+ int h) {
+ uint16x8_t sum = vdupq_n_u16(0);
+
+ int i = 0;
+ do {
+ uint8x16_t s = vld1q_u8(src_ptr);
+ uint8x16_t r = vld1q_u8(ref_ptr);
+
+ uint8x16_t diff = vabdq_u8(s, r);
+ sum = vpadalq_u8(sum, diff);
+
+ src_ptr += src_stride;
+ ref_ptr += ref_stride;
+ i++;
+ } while (i < h);
+
+ return horizontal_add_u16x8(sum);
+}
+
+#endif // defined(__ARM_FEATURE_DOTPROD)
+
static INLINE unsigned int sad8xh_neon(const uint8_t *src_ptr, int src_stride,
const uint8_t *ref_ptr, int ref_stride,
int h) {
- uint16x8_t q3 = vdupq_n_u16(0);
- for (int y = 0; y < h; y++) {
- uint8x8_t q0 = vld1_u8(src_ptr);
- uint8x8_t q1 = vld1_u8(ref_ptr);
+ uint16x8_t sum = vdupq_n_u16(0);
+
+ int i = 0;
+ do {
+ uint8x8_t s = vld1_u8(src_ptr);
+ uint8x8_t r = vld1_u8(ref_ptr);
+
+ sum = vabal_u8(sum, s, r);
+
src_ptr += src_stride;
ref_ptr += ref_stride;
- q3 = vabal_u8(q3, q0, q1);
- }
- return horizontal_add_u16x8(q3);
+ i++;
+ } while (i < h);
+
+ return horizontal_add_u16x8(sum);
}
static INLINE unsigned int sad4xh_neon(const uint8_t *src_ptr, int src_stride,
const uint8_t *ref_ptr, int ref_stride,
int h) {
- uint16x8_t q3 = vdupq_n_u16(0);
- uint32x2_t q0 = vdup_n_u32(0);
- uint32x2_t q1 = vdup_n_u32(0);
- uint32_t src4, ref4;
- for (int y = 0; y < h / 2; y++) {
- memcpy(&src4, src_ptr, 4);
- memcpy(&ref4, ref_ptr, 4);
+ uint16x8_t sum = vdupq_n_u16(0);
+
+ int i = 0;
+ do {
+ uint32x2_t s, r;
+ uint32_t s0, s1, r0, r1;
+
+ memcpy(&s0, src_ptr, 4);
+ memcpy(&r0, ref_ptr, 4);
+ s = vdup_n_u32(s0);
+ r = vdup_n_u32(r0);
src_ptr += src_stride;
ref_ptr += ref_stride;
- q0 = vset_lane_u32(src4, q0, 0);
- q1 = vset_lane_u32(ref4, q1, 0);
- memcpy(&src4, src_ptr, 4);
- memcpy(&ref4, ref_ptr, 4);
+ memcpy(&s1, src_ptr, 4);
+ memcpy(&r1, ref_ptr, 4);
+ s = vset_lane_u32(s1, s, 1);
+ r = vset_lane_u32(r1, r, 1);
src_ptr += src_stride;
ref_ptr += ref_stride;
- q0 = vset_lane_u32(src4, q0, 1);
- q1 = vset_lane_u32(ref4, q1, 1);
- q3 = vabal_u8(q3, vreinterpret_u8_u32(q0), vreinterpret_u8_u32(q1));
- }
- return horizontal_add_u16x8(q3);
+ sum = vabal_u8(sum, vreinterpret_u8_u32(s), vreinterpret_u8_u32(r));
+ i++;
+ } while (i < h / 2);
+
+ return horizontal_add_u16x8(sum);
}
-#define FSADS128_H(h) \
- unsigned int aom_sad_skip_128x##h##_neon( \
- const uint8_t *src_ptr, int src_stride, const uint8_t *ref_ptr, \
- int ref_stride) { \
- const uint32_t sum = sad128xh_neon(src_ptr, 2 * src_stride, ref_ptr, \
- 2 * ref_stride, h / 2); \
- return 2 * sum; \
+#define SAD_WXH_NEON(w, h) \
+ unsigned int aom_sad##w##x##h##_neon(const uint8_t *src, int src_stride, \
+ const uint8_t *ref, int ref_stride) { \
+ return sad##w##xh_neon(src, src_stride, ref, ref_stride, (h)); \
}
-FSADS128_H(128)
-FSADS128_H(64)
+SAD_WXH_NEON(4, 4)
+SAD_WXH_NEON(4, 8)
+SAD_WXH_NEON(4, 16)
-#undef FSADS128_H
+SAD_WXH_NEON(8, 4)
+SAD_WXH_NEON(8, 8)
+SAD_WXH_NEON(8, 16)
+SAD_WXH_NEON(8, 32)
-#define FSADS64_H(h) \
- unsigned int aom_sad_skip_64x##h##_neon( \
- const uint8_t *src_ptr, int src_stride, const uint8_t *ref_ptr, \
- int ref_stride) { \
- return 2 * sad64xh_neon(src_ptr, src_stride * 2, ref_ptr, ref_stride * 2, \
- h / 2); \
+SAD_WXH_NEON(16, 4)
+SAD_WXH_NEON(16, 8)
+SAD_WXH_NEON(16, 16)
+SAD_WXH_NEON(16, 32)
+SAD_WXH_NEON(16, 64)
+
+SAD_WXH_NEON(32, 8)
+SAD_WXH_NEON(32, 16)
+SAD_WXH_NEON(32, 32)
+SAD_WXH_NEON(32, 64)
+
+SAD_WXH_NEON(64, 16)
+SAD_WXH_NEON(64, 32)
+SAD_WXH_NEON(64, 64)
+SAD_WXH_NEON(64, 128)
+
+SAD_WXH_NEON(128, 64)
+SAD_WXH_NEON(128, 128)
+
+#undef SAD_WXH_NEON
+
+#define SAD_SKIP_WXH_NEON(w, h) \
+ unsigned int aom_sad_skip_##w##x##h##_neon( \
+ const uint8_t *src, int src_stride, const uint8_t *ref, \
+ int ref_stride) { \
+ return 2 * \
+ sad##w##xh_neon(src, 2 * src_stride, ref, 2 * ref_stride, (h) / 2); \
}
-FSADS64_H(128)
-FSADS64_H(64)
-FSADS64_H(32)
-FSADS64_H(16)
+SAD_SKIP_WXH_NEON(4, 8)
+SAD_SKIP_WXH_NEON(4, 16)
-#undef FSADS64_H
+SAD_SKIP_WXH_NEON(8, 8)
+SAD_SKIP_WXH_NEON(8, 16)
+SAD_SKIP_WXH_NEON(8, 32)
-#define FSADS32_H(h) \
- unsigned int aom_sad_skip_32x##h##_neon( \
- const uint8_t *src_ptr, int src_stride, const uint8_t *ref_ptr, \
- int ref_stride) { \
- return 2 * sad32xh_neon(src_ptr, src_stride * 2, ref_ptr, ref_stride * 2, \
- h / 2); \
- }
+SAD_SKIP_WXH_NEON(16, 8)
+SAD_SKIP_WXH_NEON(16, 16)
+SAD_SKIP_WXH_NEON(16, 32)
+SAD_SKIP_WXH_NEON(16, 64)
-FSADS32_H(64)
-FSADS32_H(32)
-FSADS32_H(16)
-FSADS32_H(8)
+SAD_SKIP_WXH_NEON(32, 8)
+SAD_SKIP_WXH_NEON(32, 16)
+SAD_SKIP_WXH_NEON(32, 32)
+SAD_SKIP_WXH_NEON(32, 64)
-#undef FSADS32_H
+SAD_SKIP_WXH_NEON(64, 16)
+SAD_SKIP_WXH_NEON(64, 32)
+SAD_SKIP_WXH_NEON(64, 64)
+SAD_SKIP_WXH_NEON(64, 128)
-#define FSADS16_H(h) \
- unsigned int aom_sad_skip_16x##h##_neon( \
- const uint8_t *src_ptr, int src_stride, const uint8_t *ref_ptr, \
- int ref_stride) { \
- return 2 * sad16xh_neon(src_ptr, src_stride * 2, ref_ptr, ref_stride * 2, \
- h / 2); \
- }
+SAD_SKIP_WXH_NEON(128, 64)
+SAD_SKIP_WXH_NEON(128, 128)
-FSADS16_H(64)
-FSADS16_H(32)
-FSADS16_H(16)
-FSADS16_H(8)
-
-#undef FSADS16_H
-
-#define FSADS8_H(h) \
- unsigned int aom_sad_skip_8x##h##_neon( \
- const uint8_t *src_ptr, int src_stride, const uint8_t *ref_ptr, \
- int ref_stride) { \
- return 2 * sad8xh_neon(src_ptr, src_stride * 2, ref_ptr, ref_stride * 2, \
- h / 2); \
- }
-
-FSADS8_H(32)
-FSADS8_H(16)
-FSADS8_H(8)
-
-#undef FSADS8_H
-
-#define FSADS4_H(h) \
- unsigned int aom_sad_skip_4x##h##_neon( \
- const uint8_t *src_ptr, int src_stride, const uint8_t *ref_ptr, \
- int ref_stride) { \
- return 2 * sad4xh_neon(src_ptr, src_stride * 2, ref_ptr, ref_stride * 2, \
- h / 2); \
- }
-
-FSADS4_H(16)
-FSADS4_H(8)
-
-#undef FSADS4_H
+#undef SAD_SKIP_WXH_NEON
diff --git a/aom_dsp/arm/sse_neon.c b/aom_dsp/arm/sse_neon.c
index a69dfb5..4370146 100644
--- a/aom_dsp/arm/sse_neon.c
+++ b/aom_dsp/arm/sse_neon.c
@@ -16,141 +16,315 @@
#include "aom_dsp/arm/sum_neon.h"
#include "aom_dsp/arm/transpose_neon.h"
-static INLINE void sse_w16_neon(uint32x4_t *sum, const uint8_t *a,
- const uint8_t *b) {
- const uint8x16_t v_a0 = vld1q_u8(a);
- const uint8x16_t v_b0 = vld1q_u8(b);
- const uint8x16_t diff = vabdq_u8(v_a0, v_b0);
- const uint8x8_t diff_lo = vget_low_u8(diff);
- const uint8x8_t diff_hi = vget_high_u8(diff);
- *sum = vpadalq_u16(*sum, vmull_u8(diff_lo, diff_lo));
- *sum = vpadalq_u16(*sum, vmull_u8(diff_hi, diff_hi));
+#if defined(__ARM_FEATURE_DOTPROD)
+
+static INLINE void sse_16x1_neon(const uint8_t *src, const uint8_t *ref,
+ uint32x4_t *sse) {
+ uint8x16_t s = vld1q_u8(src);
+ uint8x16_t r = vld1q_u8(ref);
+
+ uint8x16_t abs_diff = vabdq_u8(s, r);
+
+ *sse = vdotq_u32(*sse, abs_diff, abs_diff);
}
-static INLINE void aom_sse4x2_neon(const uint8_t *a, int a_stride,
- const uint8_t *b, int b_stride,
- uint32x4_t *sum) {
- uint8x8_t v_a0, v_b0;
- v_a0 = v_b0 = vcreate_u8(0);
- // above line is only to shadow [-Werror=uninitialized]
- v_a0 = vreinterpret_u8_u32(
- vld1_lane_u32((uint32_t *)a, vreinterpret_u32_u8(v_a0), 0));
- v_a0 = vreinterpret_u8_u32(
- vld1_lane_u32((uint32_t *)(a + a_stride), vreinterpret_u32_u8(v_a0), 1));
- v_b0 = vreinterpret_u8_u32(
- vld1_lane_u32((uint32_t *)b, vreinterpret_u32_u8(v_b0), 0));
- v_b0 = vreinterpret_u8_u32(
- vld1_lane_u32((uint32_t *)(b + b_stride), vreinterpret_u32_u8(v_b0), 1));
- const uint8x8_t v_a_w = vabd_u8(v_a0, v_b0);
- *sum = vpadalq_u16(*sum, vmull_u8(v_a_w, v_a_w));
+
+static INLINE void sse_8x1_neon(const uint8_t *src, const uint8_t *ref,
+ uint32x2_t *sse) {
+ uint8x8_t s = vld1_u8(src);
+ uint8x8_t r = vld1_u8(ref);
+
+ uint8x8_t abs_diff = vabd_u8(s, r);
+
+ *sse = vdot_u32(*sse, abs_diff, abs_diff);
}
-static INLINE void aom_sse8_neon(const uint8_t *a, const uint8_t *b,
- uint32x4_t *sum) {
- const uint8x8_t v_a_w = vld1_u8(a);
- const uint8x8_t v_b_w = vld1_u8(b);
- const uint8x8_t v_d_w = vabd_u8(v_a_w, v_b_w);
- *sum = vpadalq_u16(*sum, vmull_u8(v_d_w, v_d_w));
+
+static INLINE void sse_4x2_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ uint32x2_t *sse) {
+ uint8x8_t s = load_unaligned_u8(src, src_stride);
+ uint8x8_t r = load_unaligned_u8(ref, ref_stride);
+
+ uint8x8_t abs_diff = vabd_u8(s, r);
+
+ *sse = vdot_u32(*sse, abs_diff, abs_diff);
}
-int64_t aom_sse_neon(const uint8_t *a, int a_stride, const uint8_t *b,
- int b_stride, int width, int height) {
- int y = 0;
- int64_t sse = 0;
- uint32x4_t sum = vdupq_n_u32(0);
- switch (width) {
- case 4:
+
+static INLINE uint32_t sse_8xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ int height) {
+ uint32x2_t sse[2] = { vdup_n_u32(0), vdup_n_u32(0) };
+
+ int i = 0;
+ do {
+ sse_8x1_neon(src, ref, &sse[0]);
+ src += src_stride;
+ ref += ref_stride;
+ sse_8x1_neon(src, ref, &sse[1]);
+ src += src_stride;
+ ref += ref_stride;
+ i += 2;
+ } while (i < height);
+
+ return horizontal_add_u32x4(vcombine_u32(sse[0], sse[1]));
+}
+
+static INLINE uint32_t sse_4xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ int height) {
+ uint32x2_t sse = vdup_n_u32(0);
+
+ int i = 0;
+ do {
+ sse_4x2_neon(src, src_stride, ref, ref_stride, &sse);
+
+ src += 2 * src_stride;
+ ref += 2 * ref_stride;
+ i += 2;
+ } while (i < height);
+
+ return horizontal_add_u32x2(sse);
+}
+
+static INLINE uint32_t sse_wxh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ int width, int height) {
+ uint32x2_t sse[2] = { vdup_n_u32(0), vdup_n_u32(0) };
+
+ if ((width & 0x07) && ((width & 0x07) < 5)) {
+ int i = 0;
+ do {
+ int j = 0;
do {
- aom_sse4x2_neon(a, a_stride, b, b_stride, &sum);
- a += a_stride << 1;
- b += b_stride << 1;
- y += 2;
- } while (y < height);
- sse = horizontal_add_s32x4(vreinterpretq_s32_u32(sum));
- break;
- case 8:
+ sse_8x1_neon(src + j, ref + j, &sse[0]);
+ sse_8x1_neon(src + j + src_stride, ref + j + ref_stride, &sse[1]);
+ j += 8;
+ } while (j + 4 < width);
+
+ sse_4x2_neon(src + j, src_stride, ref + j, ref_stride, &sse[0]);
+ src += 2 * src_stride;
+ ref += 2 * ref_stride;
+ i += 2;
+ } while (i < height);
+ } else {
+ int i = 0;
+ do {
+ int j = 0;
do {
- aom_sse8_neon(a, b, &sum);
- a += a_stride;
- b += b_stride;
- y += 1;
- } while (y < height);
- sse = horizontal_add_s32x4(vreinterpretq_s32_u32(sum));
- break;
- case 16:
- do {
- sse_w16_neon(&sum, a, b);
- a += a_stride;
- b += b_stride;
- y += 1;
- } while (y < height);
- sse = horizontal_add_s32x4(vreinterpretq_s32_u32(sum));
- break;
- case 32:
- do {
- sse_w16_neon(&sum, a, b);
- sse_w16_neon(&sum, a + 16, b + 16);
- a += a_stride;
- b += b_stride;
- y += 1;
- } while (y < height);
- sse = horizontal_add_s32x4(vreinterpretq_s32_u32(sum));
- break;
- case 64:
- do {
- sse_w16_neon(&sum, a, b);
- sse_w16_neon(&sum, a + 16 * 1, b + 16 * 1);
- sse_w16_neon(&sum, a + 16 * 2, b + 16 * 2);
- sse_w16_neon(&sum, a + 16 * 3, b + 16 * 3);
- a += a_stride;
- b += b_stride;
- y += 1;
- } while (y < height);
- sse = horizontal_add_s32x4(vreinterpretq_s32_u32(sum));
- break;
- case 128:
- do {
- sse_w16_neon(&sum, a, b);
- sse_w16_neon(&sum, a + 16 * 1, b + 16 * 1);
- sse_w16_neon(&sum, a + 16 * 2, b + 16 * 2);
- sse_w16_neon(&sum, a + 16 * 3, b + 16 * 3);
- sse_w16_neon(&sum, a + 16 * 4, b + 16 * 4);
- sse_w16_neon(&sum, a + 16 * 5, b + 16 * 5);
- sse_w16_neon(&sum, a + 16 * 6, b + 16 * 6);
- sse_w16_neon(&sum, a + 16 * 7, b + 16 * 7);
- a += a_stride;
- b += b_stride;
- y += 1;
- } while (y < height);
- sse = horizontal_add_s32x4(vreinterpretq_s32_u32(sum));
- break;
- default:
- if (width & 0x07) {
- do {
- int i = 0;
- do {
- aom_sse8_neon(a + i, b + i, &sum);
- aom_sse8_neon(a + i + a_stride, b + i + b_stride, &sum);
- i += 8;
- } while (i + 4 < width);
- aom_sse4x2_neon(a + i, a_stride, b + i, b_stride, &sum);
- a += (a_stride << 1);
- b += (b_stride << 1);
- y += 2;
- } while (y < height);
- } else {
- do {
- int i = 0;
- do {
- aom_sse8_neon(a + i, b + i, &sum);
- i += 8;
- } while (i < width);
- a += a_stride;
- b += b_stride;
- y += 1;
- } while (y < height);
- }
- sse = horizontal_add_s32x4(vreinterpretq_s32_u32(sum));
- break;
+ sse_8x1_neon(src + j, ref + j, &sse[0]);
+ sse_8x1_neon(src + j + src_stride, ref + j + ref_stride, &sse[1]);
+ j += 8;
+ } while (j < width);
+
+ src += 2 * src_stride;
+ ref += 2 * ref_stride;
+ i += 2;
+ } while (i < height);
}
- return sse;
+ return horizontal_add_u32x4(vcombine_u32(sse[0], sse[1]));
+}
+
+#else // !defined(__ARM_FEATURE_DOTPROD)
+
+static INLINE void sse_16x1_neon(const uint8_t *src, const uint8_t *ref,
+ uint32x4_t *sse) {
+ uint8x16_t s = vld1q_u8(src);
+ uint8x16_t r = vld1q_u8(ref);
+
+ uint8x16_t abs_diff = vabdq_u8(s, r);
+ uint8x8_t abs_diff_lo = vget_low_u8(abs_diff);
+ uint8x8_t abs_diff_hi = vget_high_u8(abs_diff);
+
+ *sse = vpadalq_u16(*sse, vmull_u8(abs_diff_lo, abs_diff_lo));
+ *sse = vpadalq_u16(*sse, vmull_u8(abs_diff_hi, abs_diff_hi));
+}
+
+static INLINE void sse_8x1_neon(const uint8_t *src, const uint8_t *ref,
+ uint32x4_t *sse) {
+ uint8x8_t s = vld1_u8(src);
+ uint8x8_t r = vld1_u8(ref);
+
+ uint8x8_t abs_diff = vabd_u8(s, r);
+
+ *sse = vpadalq_u16(*sse, vmull_u8(abs_diff, abs_diff));
+}
+
+static INLINE void sse_4x2_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ uint32x4_t *sse) {
+ uint8x8_t s = load_unaligned_u8(src, src_stride);
+ uint8x8_t r = load_unaligned_u8(ref, ref_stride);
+
+ uint8x8_t abs_diff = vabd_u8(s, r);
+
+ *sse = vpadalq_u16(*sse, vmull_u8(abs_diff, abs_diff));
+}
+
+static INLINE uint32_t sse_8xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ int height) {
+ uint32x4_t sse = vdupq_n_u32(0);
+
+ int i = 0;
+ do {
+ sse_8x1_neon(src, ref, &sse);
+
+ src += src_stride;
+ ref += ref_stride;
+ i++;
+ } while (i < height);
+
+ return horizontal_add_u32x4(sse);
+}
+
+static INLINE uint32_t sse_4xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ int height) {
+ uint32x4_t sse = vdupq_n_u32(0);
+
+ int i = 0;
+ do {
+ sse_4x2_neon(src, src_stride, ref, ref_stride, &sse);
+
+ src += 2 * src_stride;
+ ref += 2 * ref_stride;
+ i += 2;
+ } while (i < height);
+
+ return horizontal_add_u32x4(sse);
+}
+
+static INLINE uint32_t sse_wxh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ int width, int height) {
+ uint32x4_t sse = vdupq_n_u32(0);
+
+ if ((width & 0x07) && ((width & 0x07) < 5)) {
+ int i = 0;
+ do {
+ int j = 0;
+ do {
+ sse_8x1_neon(src + j, ref + j, &sse);
+ sse_8x1_neon(src + j + src_stride, ref + j + ref_stride, &sse);
+ j += 8;
+ } while (j + 4 < width);
+
+ sse_4x2_neon(src + j, src_stride, ref + j, ref_stride, &sse);
+ src += 2 * src_stride;
+ ref += 2 * ref_stride;
+ i += 2;
+ } while (i < height);
+ } else {
+ int i = 0;
+ do {
+ int j = 0;
+ do {
+ sse_8x1_neon(src + j, ref + j, &sse);
+ j += 8;
+ } while (j < width);
+
+ src += src_stride;
+ ref += ref_stride;
+ i++;
+ } while (i < height);
+ }
+ return horizontal_add_u32x4(sse);
+}
+
+#endif // defined(__ARM_FEATURE_DOTPROD)
+
+static INLINE uint32_t sse_128xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ int height) {
+ uint32x4_t sse[2] = { vdupq_n_u32(0), vdupq_n_u32(0) };
+
+ int i = 0;
+ do {
+ sse_16x1_neon(src, ref, &sse[0]);
+ sse_16x1_neon(src + 16, ref + 16, &sse[1]);
+ sse_16x1_neon(src + 32, ref + 32, &sse[0]);
+ sse_16x1_neon(src + 48, ref + 48, &sse[1]);
+ sse_16x1_neon(src + 64, ref + 64, &sse[0]);
+ sse_16x1_neon(src + 80, ref + 80, &sse[1]);
+ sse_16x1_neon(src + 96, ref + 96, &sse[0]);
+ sse_16x1_neon(src + 112, ref + 112, &sse[1]);
+
+ src += src_stride;
+ ref += ref_stride;
+ i++;
+ } while (i < height);
+
+ return horizontal_add_u32x4(vaddq_u32(sse[0], sse[1]));
+}
+
+static INLINE uint32_t sse_64xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ int height) {
+ uint32x4_t sse[2] = { vdupq_n_u32(0), vdupq_n_u32(0) };
+
+ int i = 0;
+ do {
+ sse_16x1_neon(src, ref, &sse[0]);
+ sse_16x1_neon(src + 16, ref + 16, &sse[1]);
+ sse_16x1_neon(src + 32, ref + 32, &sse[0]);
+ sse_16x1_neon(src + 48, ref + 48, &sse[1]);
+
+ src += src_stride;
+ ref += ref_stride;
+ i++;
+ } while (i < height);
+
+ return horizontal_add_u32x4(vaddq_u32(sse[0], sse[1]));
+}
+
+static INLINE uint32_t sse_32xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ int height) {
+ uint32x4_t sse[2] = { vdupq_n_u32(0), vdupq_n_u32(0) };
+
+ int i = 0;
+ do {
+ sse_16x1_neon(src, ref, &sse[0]);
+ sse_16x1_neon(src + 16, ref + 16, &sse[1]);
+
+ src += src_stride;
+ ref += ref_stride;
+ i++;
+ } while (i < height);
+
+ return horizontal_add_u32x4(vaddq_u32(sse[0], sse[1]));
+}
+
+static INLINE uint32_t sse_16xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ int height) {
+ uint32x4_t sse[2] = { vdupq_n_u32(0), vdupq_n_u32(0) };
+
+ int i = 0;
+ do {
+ sse_16x1_neon(src, ref, &sse[0]);
+ src += src_stride;
+ ref += ref_stride;
+ sse_16x1_neon(src, ref, &sse[1]);
+ src += src_stride;
+ ref += ref_stride;
+ i += 2;
+ } while (i < height);
+
+ return horizontal_add_u32x4(vaddq_u32(sse[0], sse[1]));
+}
+
+int64_t aom_sse_neon(const uint8_t *src, int src_stride, const uint8_t *ref,
+ int ref_stride, int width, int height) {
+ switch (width) {
+ case 4: return sse_4xh_neon(src, src_stride, ref, ref_stride, height);
+ case 8: return sse_8xh_neon(src, src_stride, ref, ref_stride, height);
+ case 16: return sse_16xh_neon(src, src_stride, ref, ref_stride, height);
+ case 32: return sse_32xh_neon(src, src_stride, ref, ref_stride, height);
+ case 64: return sse_64xh_neon(src, src_stride, ref, ref_stride, height);
+ case 128: return sse_128xh_neon(src, src_stride, ref, ref_stride, height);
+ default:
+ return sse_wxh_neon(src, src_stride, ref, ref_stride, width, height);
+ }
}
#if CONFIG_AV1_HIGHBITDEPTH
diff --git a/aom_dsp/arm/subpel_variance_neon.c b/aom_dsp/arm/subpel_variance_neon.c
index 4ecf891..4615038 100644
--- a/aom_dsp/arm/subpel_variance_neon.c
+++ b/aom_dsp/arm/subpel_variance_neon.c
@@ -17,424 +17,233 @@
#include "aom_ports/mem.h"
#include "aom/aom_integer.h"
-#include "aom_dsp/aom_filter.h"
#include "aom_dsp/variance.h"
+#include "aom_dsp/arm/mem_neon.h"
-// Load 2 sets of 4 bytes when alignment is not guaranteed.
-static INLINE uint8x8_t load_unaligned_u8(const uint8_t *buf, int stride) {
- uint32_t a;
- uint32x2_t a_u32 = vdup_n_u32(0);
- if (stride == 4) return vld1_u8(buf);
- memcpy(&a, buf, 4);
- buf += stride;
- a_u32 = vld1_lane_u32(&a, a_u32, 0);
- memcpy(&a, buf, 4);
- a_u32 = vld1_lane_u32(&a, a_u32, 1);
- return vreinterpret_u8_u32(a_u32);
+static void var_filter_block2d_bil_w4(const uint8_t *src_ptr, uint8_t *dst_ptr,
+ int src_stride, int pixel_step,
+ int dst_height, int filter_offset) {
+ const uint8x8_t f0 = vdup_n_u8(8 - filter_offset);
+ const uint8x8_t f1 = vdup_n_u8(filter_offset);
+
+ int i = 0;
+ do {
+ uint8x8_t s0 = load_unaligned_u8(src_ptr, src_stride);
+ uint8x8_t s1 = load_unaligned_u8(src_ptr + pixel_step, src_stride);
+ uint16x8_t blend = vmull_u8(s0, f0);
+ blend = vmlal_u8(blend, s1, f1);
+ uint8x8_t blend_u8 = vrshrn_n_u16(blend, 3);
+ vst1_u8(dst_ptr, blend_u8);
+
+ src_ptr += 2 * src_stride;
+ dst_ptr += 2 * 4;
+ i += 2;
+ } while (i < dst_height);
}
-// Process a block exactly 4 wide and a multiple of 2 high.
-static void var_filter_block2d_bil_w4(const uint8_t *src_ptr,
- uint8_t *output_ptr,
- unsigned int src_pixels_per_line,
- int pixel_step,
- unsigned int output_height,
- const uint8_t *filter) {
- const uint8x8_t f0 = vdup_n_u8(filter[0]);
- const uint8x8_t f1 = vdup_n_u8(filter[1]);
- unsigned int i;
- for (i = 0; i < output_height; i += 2) {
- const uint8x8_t src_0 = load_unaligned_u8(src_ptr, src_pixels_per_line);
- const uint8x8_t src_1 =
- load_unaligned_u8(src_ptr + pixel_step, src_pixels_per_line);
- const uint16x8_t a = vmull_u8(src_0, f0);
- const uint16x8_t b = vmlal_u8(a, src_1, f1);
- const uint8x8_t out = vrshrn_n_u16(b, FILTER_BITS);
- vst1_u8(output_ptr, out);
- src_ptr += 2 * src_pixels_per_line;
- output_ptr += 8;
+static void var_filter_block2d_bil_w8(const uint8_t *src_ptr, uint8_t *dst_ptr,
+ int src_stride, int pixel_step,
+ int dst_height, int filter_offset) {
+ const uint8x8_t f0 = vdup_n_u8(8 - filter_offset);
+ const uint8x8_t f1 = vdup_n_u8(filter_offset);
+
+ int i = 0;
+ do {
+ uint8x8_t s0 = vld1_u8(src_ptr);
+ uint8x8_t s1 = vld1_u8(src_ptr + pixel_step);
+ uint16x8_t blend = vmull_u8(s0, f0);
+ blend = vmlal_u8(blend, s1, f1);
+ uint8x8_t blend_u8 = vrshrn_n_u16(blend, 3);
+ vst1_u8(dst_ptr, blend_u8);
+
+ src_ptr += src_stride;
+ dst_ptr += 8;
+ i++;
+ } while (i < dst_height);
+}
+
+static void var_filter_block2d_bil_large(const uint8_t *src_ptr,
+ uint8_t *dst_ptr, int src_stride,
+ int pixel_step, int dst_width,
+ int dst_height, int filter_offset) {
+ const uint8x8_t f0 = vdup_n_u8(8 - filter_offset);
+ const uint8x8_t f1 = vdup_n_u8(filter_offset);
+
+ int i = 0;
+ do {
+ int j = 0;
+ do {
+ uint8x16_t s0 = vld1q_u8(src_ptr + j);
+ uint8x16_t s1 = vld1q_u8(src_ptr + j + pixel_step);
+ uint16x8_t blend_l = vmull_u8(vget_low_u8(s0), f0);
+ blend_l = vmlal_u8(blend_l, vget_low_u8(s1), f1);
+ uint16x8_t blend_h = vmull_u8(vget_high_u8(s0), f0);
+ blend_h = vmlal_u8(blend_h, vget_high_u8(s1), f1);
+ uint8x16_t blend_u8 =
+ vcombine_u8(vrshrn_n_u16(blend_l, 3), vrshrn_n_u16(blend_h, 3));
+ vst1q_u8(dst_ptr + j, blend_u8);
+
+ j += 16;
+ } while (j < dst_width);
+
+ src_ptr += src_stride;
+ dst_ptr += dst_width;
+ i++;
+ } while (i < dst_height);
+}
+
+static void var_filter_block2d_bil_w16(const uint8_t *src_ptr, uint8_t *dst_ptr,
+ int src_stride, int pixel_step,
+ int dst_height, int filter_offset) {
+ var_filter_block2d_bil_large(src_ptr, dst_ptr, src_stride, pixel_step, 16,
+ dst_height, filter_offset);
+}
+
+static void var_filter_block2d_bil_w32(const uint8_t *src_ptr, uint8_t *dst_ptr,
+ int src_stride, int pixel_step,
+ int dst_height, int filter_offset) {
+ var_filter_block2d_bil_large(src_ptr, dst_ptr, src_stride, pixel_step, 32,
+ dst_height, filter_offset);
+}
+
+static void var_filter_block2d_bil_w64(const uint8_t *src_ptr, uint8_t *dst_ptr,
+ int src_stride, int pixel_step,
+ int dst_height, int filter_offset) {
+ var_filter_block2d_bil_large(src_ptr, dst_ptr, src_stride, pixel_step, 64,
+ dst_height, filter_offset);
+}
+
+static void var_filter_block2d_bil_w128(const uint8_t *src_ptr,
+ uint8_t *dst_ptr, int src_stride,
+ int pixel_step, int dst_height,
+ int filter_offset) {
+ var_filter_block2d_bil_large(src_ptr, dst_ptr, src_stride, pixel_step, 128,
+ dst_height, filter_offset);
+}
+
+static void var_filter_block2d_avg(const uint8_t *src_ptr, uint8_t *dst_ptr,
+ int src_stride, int pixel_step,
+ int dst_width, int dst_height) {
+ // We only specialise on the filter values for large block sizes (>= 16x16.)
+ assert(dst_width >= 16 && dst_width % 16 == 0);
+
+ int i = 0;
+ do {
+ int j = 0;
+ do {
+ uint8x16_t s0 = vld1q_u8(src_ptr + j);
+ uint8x16_t s1 = vld1q_u8(src_ptr + j + pixel_step);
+ uint8x16_t avg = vrhaddq_u8(s0, s1);
+ vst1q_u8(dst_ptr + j, avg);
+
+ j += 16;
+ } while (j < dst_width);
+
+ src_ptr += src_stride;
+ dst_ptr += dst_width;
+ i++;
+ } while (i < dst_height);
+}
+
+#define SUBPEL_VARIANCE_WXH_NEON(w, h, padding) \
+ unsigned int aom_sub_pixel_variance##w##x##h##_neon( \
+ const uint8_t *src, int src_stride, int xoffset, int yoffset, \
+ const uint8_t *ref, int ref_stride, uint32_t *sse) { \
+ uint8_t tmp0[w * (h + padding)]; \
+ uint8_t tmp1[w * h]; \
+ var_filter_block2d_bil_w##w(src, tmp0, src_stride, 1, (h + padding), \
+ xoffset); \
+ var_filter_block2d_bil_w##w(tmp0, tmp1, w, w, h, yoffset); \
+ return aom_variance##w##x##h(tmp1, w, ref, ref_stride, sse); \
}
-}
-static void var_filter_block2d_bil_w8(const uint8_t *src_ptr,
- uint8_t *output_ptr,
- unsigned int src_pixels_per_line,
- int pixel_step,
- unsigned int output_height,
- unsigned int output_width,
- const uint8_t *filter) {
- const uint8x8_t f0 = vdup_n_u8(filter[0]);
- const uint8x8_t f1 = vdup_n_u8(filter[1]);
- unsigned int i;
- for (i = 0; i < output_height; ++i) {
- const uint8x8_t src_0 = vld1_u8(&src_ptr[0]);
- const uint8x8_t src_1 = vld1_u8(&src_ptr[pixel_step]);
- const uint16x8_t a = vmull_u8(src_0, f0);
- const uint16x8_t b = vmlal_u8(a, src_1, f1);
- const uint8x8_t out = vrshrn_n_u16(b, FILTER_BITS);
- vst1_u8(output_ptr, out);
- // Next row...
- src_ptr += src_pixels_per_line;
- output_ptr += output_width;
+#define SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON(w, h, padding) \
+ unsigned int aom_sub_pixel_variance##w##x##h##_neon( \
+ const uint8_t *src, int src_stride, int xoffset, int yoffset, \
+ const uint8_t *ref, int ref_stride, unsigned int *sse) { \
+ if (xoffset == 0) { \
+ if (yoffset == 0) { \
+ return aom_variance##w##x##h##_neon(src, src_stride, ref, ref_stride, \
+ sse); \
+ } else if (yoffset == 4) { \
+ uint8_t tmp[w * h]; \
+ var_filter_block2d_avg(src, tmp, src_stride, src_stride, w, h); \
+ return aom_variance##w##x##h##_neon(tmp, w, ref, ref_stride, sse); \
+ } else { \
+ uint8_t tmp[w * h]; \
+ var_filter_block2d_bil_w##w(src, tmp, src_stride, src_stride, h, \
+ yoffset); \
+ return aom_variance##w##x##h##_neon(tmp, w, ref, ref_stride, sse); \
+ } \
+ } else if (xoffset == 4) { \
+ uint8_t tmp0[w * (h + padding)]; \
+ var_filter_block2d_avg(src, tmp0, src_stride, 1, w, h + padding); \
+ if (yoffset == 0) { \
+ return aom_variance##w##x##h##_neon(tmp0, w, ref, ref_stride, sse); \
+ } else if (yoffset == 4) { \
+ uint8_t tmp1[w * (h + padding)]; \
+ var_filter_block2d_avg(tmp0, tmp1, w, w, w, h); \
+ return aom_variance##w##x##h##_neon(tmp1, w, ref, ref_stride, sse); \
+ } else { \
+ uint8_t tmp1[w * (h + padding)]; \
+ var_filter_block2d_bil_w##w(tmp0, tmp1, w, w, h, yoffset); \
+ return aom_variance##w##x##h##_neon(tmp1, w, ref, ref_stride, sse); \
+ } \
+ } else { \
+ uint8_t tmp0[w * (h + padding)]; \
+ var_filter_block2d_bil_w##w(src, tmp0, src_stride, 1, (h + padding), \
+ xoffset); \
+ if (yoffset == 0) { \
+ return aom_variance##w##x##h##_neon(tmp0, w, ref, ref_stride, sse); \
+ } else if (yoffset == 4) { \
+ uint8_t tmp1[w * h]; \
+ var_filter_block2d_avg(tmp0, tmp1, w, w, w, h); \
+ return aom_variance##w##x##h##_neon(tmp1, w, ref, ref_stride, sse); \
+ } else { \
+ uint8_t tmp1[w * h]; \
+ var_filter_block2d_bil_w##w(tmp0, tmp1, w, w, h, yoffset); \
+ return aom_variance##w##x##h##_neon(tmp1, w, ref, ref_stride, sse); \
+ } \
+ } \
}
-}
-// Process a block which is a mutiple of 16 wide and any height.
-static void var_filter_block2d_bil_w16(const uint8_t *src_ptr,
- uint8_t *output_ptr,
- unsigned int src_pixels_per_line,
- int pixel_step,
- unsigned int output_height,
- unsigned int output_width,
- const uint8_t *filter) {
- const uint8x8_t f0 = vdup_n_u8(filter[0]);
- const uint8x8_t f1 = vdup_n_u8(filter[1]);
- unsigned int i, j;
- for (i = 0; i < output_height; ++i) {
- for (j = 0; j < output_width; j += 16) {
- const uint8x16_t src_0 = vld1q_u8(&src_ptr[j]);
- const uint8x16_t src_1 = vld1q_u8(&src_ptr[j + pixel_step]);
- const uint16x8_t a = vmull_u8(vget_low_u8(src_0), f0);
- const uint16x8_t b = vmlal_u8(a, vget_low_u8(src_1), f1);
- const uint8x8_t out_lo = vrshrn_n_u16(b, FILTER_BITS);
- const uint16x8_t c = vmull_u8(vget_high_u8(src_0), f0);
- const uint16x8_t d = vmlal_u8(c, vget_high_u8(src_1), f1);
- const uint8x8_t out_hi = vrshrn_n_u16(d, FILTER_BITS);
- vst1q_u8(output_ptr + j, vcombine_u8(out_lo, out_hi));
- }
- src_ptr += src_pixels_per_line;
- output_ptr += output_width;
- }
-}
+SUBPEL_VARIANCE_WXH_NEON(4, 4, 2)
+SUBPEL_VARIANCE_WXH_NEON(4, 8, 2)
-unsigned int aom_sub_pixel_variance8x8_neon(const uint8_t *src, int src_stride,
- int xoffset, int yoffset,
- const uint8_t *dst, int dst_stride,
- unsigned int *sse) {
- DECLARE_ALIGNED(16, uint8_t, temp2[8 * 8]);
- DECLARE_ALIGNED(16, uint8_t, fdata3[9 * 8]);
+SUBPEL_VARIANCE_WXH_NEON(8, 4, 1)
+SUBPEL_VARIANCE_WXH_NEON(8, 8, 1)
+SUBPEL_VARIANCE_WXH_NEON(8, 16, 1)
- var_filter_block2d_bil_w8(src, fdata3, src_stride, 1, 9, 8,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w8(fdata3, temp2, 8, 8, 8, 8,
- bilinear_filters_2t[yoffset]);
- return aom_variance8x8_neon(temp2, 8, dst, dst_stride, sse);
-}
+SUBPEL_VARIANCE_WXH_NEON(16, 8, 1)
+SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON(16, 16, 1)
+SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON(16, 32, 1)
-unsigned int aom_sub_pixel_variance16x16_neon(const uint8_t *src,
- int src_stride, int xoffset,
- int yoffset, const uint8_t *dst,
- int dst_stride,
- unsigned int *sse) {
- DECLARE_ALIGNED(16, uint8_t, temp2[16 * 16]);
- DECLARE_ALIGNED(16, uint8_t, fdata3[17 * 16]);
+SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON(32, 16, 1)
+SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON(32, 32, 1)
+SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON(32, 64, 1)
- var_filter_block2d_bil_w16(src, fdata3, src_stride, 1, 17, 16,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(fdata3, temp2, 16, 16, 16, 16,
- bilinear_filters_2t[yoffset]);
- return aom_variance16x16_neon(temp2, 16, dst, dst_stride, sse);
-}
+SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON(64, 32, 1)
+SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON(64, 64, 1)
+SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON(64, 128, 1)
-unsigned int aom_sub_pixel_variance32x32_neon(const uint8_t *src,
- int src_stride, int xoffset,
- int yoffset, const uint8_t *dst,
- int dst_stride,
- unsigned int *sse) {
- DECLARE_ALIGNED(16, uint8_t, temp2[32 * 32]);
- DECLARE_ALIGNED(16, uint8_t, fdata3[33 * 32]);
-
- var_filter_block2d_bil_w16(src, fdata3, src_stride, 1, 33, 32,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(fdata3, temp2, 32, 32, 32, 32,
- bilinear_filters_2t[yoffset]);
- return aom_variance32x32_neon(temp2, 32, dst, dst_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance64x64_neon(const uint8_t *src,
- int src_stride, int xoffset,
- int yoffset, const uint8_t *dst,
- int dst_stride,
- unsigned int *sse) {
- DECLARE_ALIGNED(16, uint8_t, temp2[64 * 64]);
- DECLARE_ALIGNED(16, uint8_t, fdata3[65 * 64]);
-
- var_filter_block2d_bil_w16(src, fdata3, src_stride, 1, 65, 64,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(fdata3, temp2, 64, 64, 64, 64,
- bilinear_filters_2t[yoffset]);
- return aom_variance64x64_neon(temp2, 64, dst, dst_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance4x4_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[4 * (4 + 2)];
- uint8_t temp1[4 * 4];
-
- var_filter_block2d_bil_w4(a, temp0, a_stride, 1, (4 + 2),
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w4(temp0, temp1, 4, 4, 4,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance4x4(temp1, 4, b, b_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance4x8_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[4 * (8 + 2)];
- uint8_t temp1[4 * 8];
-
- var_filter_block2d_bil_w4(a, temp0, a_stride, 1, (8 + 2),
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w4(temp0, temp1, 4, 4, 8,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance4x8(temp1, 4, b, b_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance8x4_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[8 * (4 + 1)];
- uint8_t temp1[8 * 4];
-
- var_filter_block2d_bil_w8(a, temp0, a_stride, 1, (4 + 1), 8,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w8(temp0, temp1, 8, 8, 4, 8,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance8x4(temp1, 8, b, b_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance8x16_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[8 * (16 + 1)];
- uint8_t temp1[8 * 16];
-
- var_filter_block2d_bil_w8(a, temp0, a_stride, 1, (16 + 1), 8,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w8(temp0, temp1, 8, 8, 16, 8,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance8x16(temp1, 8, b, b_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance16x8_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[16 * (8 + 1)];
- uint8_t temp1[16 * 8];
-
- var_filter_block2d_bil_w16(a, temp0, a_stride, 1, (8 + 1), 16,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(temp0, temp1, 16, 16, 8, 16,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance16x8(temp1, 16, b, b_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance16x32_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[16 * (32 + 1)];
- uint8_t temp1[16 * 32];
-
- var_filter_block2d_bil_w16(a, temp0, a_stride, 1, (32 + 1), 16,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(temp0, temp1, 16, 16, 32, 16,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance16x32(temp1, 16, b, b_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance32x16_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[32 * (16 + 1)];
- uint8_t temp1[32 * 16];
-
- var_filter_block2d_bil_w16(a, temp0, a_stride, 1, (16 + 1), 32,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(temp0, temp1, 32, 32, 16, 32,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance32x16(temp1, 32, b, b_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance32x64_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[32 * (64 + 1)];
- uint8_t temp1[32 * 64];
-
- var_filter_block2d_bil_w16(a, temp0, a_stride, 1, (64 + 1), 32,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(temp0, temp1, 32, 32, 64, 32,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance32x64(temp1, 32, b, b_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance64x32_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[64 * (32 + 1)];
- uint8_t temp1[64 * 32];
-
- var_filter_block2d_bil_w16(a, temp0, a_stride, 1, (32 + 1), 64,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(temp0, temp1, 64, 64, 32, 64,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance64x32(temp1, 64, b, b_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance64x128_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[64 * (128 + 1)];
- uint8_t temp1[64 * 128];
-
- var_filter_block2d_bil_w16(a, temp0, a_stride, 1, (128 + 1), 64,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(temp0, temp1, 64, 64, 128, 64,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance64x128(temp1, 64, b, b_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance128x64_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[128 * (64 + 1)];
- uint8_t temp1[128 * 64];
-
- var_filter_block2d_bil_w16(a, temp0, a_stride, 1, (64 + 1), 128,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(temp0, temp1, 128, 128, 64, 128,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance128x64(temp1, 128, b, b_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance128x128_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[128 * (128 + 1)];
- uint8_t temp1[128 * 128];
-
- var_filter_block2d_bil_w16(a, temp0, a_stride, 1, (128 + 1), 128,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(temp0, temp1, 128, 128, 128, 128,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance128x128(temp1, 128, b, b_stride, sse);
-}
+SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON(128, 64, 1)
+SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON(128, 128, 1)
// Realtime mode doesn't use 4x rectangular blocks.
#if !CONFIG_REALTIME_ONLY
-unsigned int aom_sub_pixel_variance4x16_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[4 * (16 + 2)];
- uint8_t temp1[4 * 16];
- var_filter_block2d_bil_w4(a, temp0, a_stride, 1, (16 + 2),
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w4(temp0, temp1, 4, 4, 16,
- bilinear_filters_2t[yoffset]);
+SUBPEL_VARIANCE_WXH_NEON(4, 16, 2)
- return aom_variance4x16(temp1, 4, b, b_stride, sse);
-}
+SUBPEL_VARIANCE_WXH_NEON(8, 32, 1)
-unsigned int aom_sub_pixel_variance8x32_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[8 * (32 + 1)];
- uint8_t temp1[8 * 32];
+SUBPEL_VARIANCE_WXH_NEON(16, 4, 1)
+SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON(16, 64, 1)
- var_filter_block2d_bil_w8(a, temp0, a_stride, 1, (32 + 1), 8,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w8(temp0, temp1, 8, 8, 32, 8,
- bilinear_filters_2t[yoffset]);
+SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON(32, 8, 1)
- return aom_variance8x32(temp1, 8, b, b_stride, sse);
-}
+SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON(64, 16, 1)
-unsigned int aom_sub_pixel_variance16x4_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[16 * (4 + 1)];
- uint8_t temp1[16 * 4];
-
- var_filter_block2d_bil_w16(a, temp0, a_stride, 1, (4 + 1), 16,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(temp0, temp1, 16, 16, 4, 16,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance16x4(temp1, 16, b, b_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance64x16_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[64 * (16 + 1)];
- uint8_t temp1[64 * 16];
-
- var_filter_block2d_bil_w16(a, temp0, a_stride, 1, (16 + 1), 64,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(temp0, temp1, 64, 64, 16, 64,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance64x16(temp1, 64, b, b_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance16x64_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[16 * (64 + 1)];
- uint8_t temp1[16 * 64];
-
- var_filter_block2d_bil_w16(a, temp0, a_stride, 1, (64 + 1), 16,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(temp0, temp1, 16, 16, 64, 16,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance16x64(temp1, 16, b, b_stride, sse);
-}
-
-unsigned int aom_sub_pixel_variance32x8_neon(const uint8_t *a, int a_stride,
- int xoffset, int yoffset,
- const uint8_t *b, int b_stride,
- uint32_t *sse) {
- uint8_t temp0[32 * (8 + 1)];
- uint8_t temp1[32 * 8];
-
- var_filter_block2d_bil_w16(a, temp0, a_stride, 1, (8 + 1), 32,
- bilinear_filters_2t[xoffset]);
- var_filter_block2d_bil_w16(temp0, temp1, 32, 32, 8, 32,
- bilinear_filters_2t[yoffset]);
-
- return aom_variance32x8(temp1, 32, b, b_stride, sse);
-}
#endif // !CONFIG_REALTIME_ONLY
+
+#undef SUBPEL_VARIANCE_WXH_NEON
+#undef SPECIALIZED_SUBPEL_VARIANCE_WXH_NEON
diff --git a/aom_dsp/arm/sum_neon.h b/aom_dsp/arm/sum_neon.h
index a118f3c..855edf6 100644
--- a/aom_dsp/arm/sum_neon.h
+++ b/aom_dsp/arm/sum_neon.h
@@ -37,6 +37,34 @@
#endif
}
+static INLINE uint64_t horizontal_add_u64x2(const uint64x2_t a) {
+#if defined(__aarch64__)
+ return vaddvq_u64(a);
+#else
+ return vgetq_lane_u64(a, 0) + vgetq_lane_u64(a, 1);
+#endif
+}
+
+static INLINE uint64_t horizontal_long_add_u32x4(const uint32x4_t a) {
+#if defined(__aarch64__)
+ return vaddlvq_u32(a);
+#else
+ const uint64x2_t b = vpaddlq_u32(a);
+ return vgetq_lane_u64(b, 0) + vgetq_lane_u64(b, 1);
+#endif
+}
+
+static INLINE unsigned int horizontal_add_u32x4(const uint32x4_t a) {
+#if defined(__aarch64__)
+ return vaddvq_u32(a);
+#else
+ const uint64x2_t b = vpaddlq_u32(a);
+ const uint32x2_t c = vadd_u32(vreinterpret_u32_u64(vget_low_u64(b)),
+ vreinterpret_u32_u64(vget_high_u64(b)));
+ return vget_lane_u32(c, 0);
+#endif
+}
+
static INLINE uint32_t horizontal_long_add_u16x8(const uint16x8_t vec_lo,
const uint16x8_t vec_hi) {
#if defined(__aarch64__)
@@ -66,6 +94,15 @@
#endif
}
+static INLINE uint32_t horizontal_add_u32x2(const uint32x2_t a) {
+#if defined(__aarch64__)
+ return vaddv_u32(a);
+#else
+ const uint64x1_t b = vpaddl_u32(a);
+ return vget_lane_u32(vreinterpret_u32_u64(b), 0);
+#endif
+}
+
static INLINE uint32_t horizontal_add_u16x4(const uint16x4_t a) {
#if defined(__aarch64__)
return vaddlv_u16(a);
diff --git a/aom_dsp/arm/sum_squares_neon.c b/aom_dsp/arm/sum_squares_neon.c
index 0b7337a..bf212a9 100644
--- a/aom_dsp/arm/sum_squares_neon.c
+++ b/aom_dsp/arm/sum_squares_neon.c
@@ -13,111 +13,83 @@
#include <assert.h>
#include "aom_dsp/arm/mem_neon.h"
+#include "aom_dsp/arm/sum_neon.h"
#include "config/aom_dsp_rtcd.h"
-static INLINE uint32x4_t sum_squares_i16_4x4_neon(const int16_t *src,
- int stride) {
- const int16x4_t v_val_01_lo = vld1_s16(src + 0 * stride);
- const int16x4_t v_val_01_hi = vld1_s16(src + 1 * stride);
- const int16x4_t v_val_23_lo = vld1_s16(src + 2 * stride);
- const int16x4_t v_val_23_hi = vld1_s16(src + 3 * stride);
- int32x4_t v_sq_01_d = vmull_s16(v_val_01_lo, v_val_01_lo);
- v_sq_01_d = vmlal_s16(v_sq_01_d, v_val_01_hi, v_val_01_hi);
- int32x4_t v_sq_23_d = vmull_s16(v_val_23_lo, v_val_23_lo);
- v_sq_23_d = vmlal_s16(v_sq_23_d, v_val_23_hi, v_val_23_hi);
-#if defined(__aarch64__)
- return vreinterpretq_u32_s32(vpaddq_s32(v_sq_01_d, v_sq_23_d));
-#else
- return vreinterpretq_u32_s32(vcombine_s32(
- vqmovn_s64(vpaddlq_s32(v_sq_01_d)), vqmovn_s64(vpaddlq_s32(v_sq_23_d))));
-#endif
+static INLINE uint64_t aom_sum_squares_2d_i16_4x4_neon(const int16_t *src,
+ int stride) {
+ int16x4_t s0 = vld1_s16(src + 0 * stride);
+ int16x4_t s1 = vld1_s16(src + 1 * stride);
+ int16x4_t s2 = vld1_s16(src + 2 * stride);
+ int16x4_t s3 = vld1_s16(src + 3 * stride);
+
+ int32x4_t sum_squares = vmull_s16(s0, s0);
+ sum_squares = vmlal_s16(sum_squares, s1, s1);
+ sum_squares = vmlal_s16(sum_squares, s2, s2);
+ sum_squares = vmlal_s16(sum_squares, s3, s3);
+
+ return horizontal_long_add_u32x4(vreinterpretq_u32_s32(sum_squares));
}
-uint64_t aom_sum_squares_2d_i16_4x4_neon(const int16_t *src, int stride) {
- const uint32x4_t v_sum_0123_d = sum_squares_i16_4x4_neon(src, stride);
-#if defined(__aarch64__)
- return (uint64_t)vaddvq_u32(v_sum_0123_d);
-#else
- uint64x2_t v_sum_d = vpaddlq_u32(v_sum_0123_d);
- v_sum_d = vaddq_u64(v_sum_d, vextq_u64(v_sum_d, v_sum_d, 1));
- return vgetq_lane_u64(v_sum_d, 0);
-#endif
-}
+static INLINE uint64_t aom_sum_squares_2d_i16_4xn_neon(const int16_t *src,
+ int stride, int height) {
+ int32x4_t sum_squares[2] = { vdupq_n_s32(0), vdupq_n_s32(0) };
-uint64_t aom_sum_squares_2d_i16_4xn_neon(const int16_t *src, int stride,
- int height) {
- int r = 0;
- uint32x4_t v_acc_q = vdupq_n_u32(0);
+ int h = 0;
do {
- const uint32x4_t v_acc_d = sum_squares_i16_4x4_neon(src, stride);
- v_acc_q = vaddq_u32(v_acc_q, v_acc_d);
- src += stride << 2;
- r += 4;
- } while (r < height);
+ int16x4_t s0 = vld1_s16(src + 0 * stride);
+ int16x4_t s1 = vld1_s16(src + 1 * stride);
+ int16x4_t s2 = vld1_s16(src + 2 * stride);
+ int16x4_t s3 = vld1_s16(src + 3 * stride);
- uint64x2_t v_acc_64 = vpaddlq_u32(v_acc_q);
-#if defined(__aarch64__)
- return vaddvq_u64(v_acc_64);
-#else
- v_acc_64 = vaddq_u64(v_acc_64, vextq_u64(v_acc_64, v_acc_64, 1));
- return vgetq_lane_u64(v_acc_64, 0);
-#endif
-}
-
-uint64_t aom_sum_squares_2d_i16_nxn_neon(const int16_t *src, int stride,
- int width, int height) {
- int r = 0;
- const int32x4_t zero = vdupq_n_s32(0);
- uint64x2_t v_acc_q = vreinterpretq_u64_s32(zero);
- do {
- int32x4_t v_sum = zero;
- int c = 0;
- do {
- const int16_t *b = src + c;
- const int16x8_t v_val_0 = vld1q_s16(b + 0 * stride);
- const int16x8_t v_val_1 = vld1q_s16(b + 1 * stride);
- const int16x8_t v_val_2 = vld1q_s16(b + 2 * stride);
- const int16x8_t v_val_3 = vld1q_s16(b + 3 * stride);
- const int16x4_t v_val_0_lo = vget_low_s16(v_val_0);
- const int16x4_t v_val_1_lo = vget_low_s16(v_val_1);
- const int16x4_t v_val_2_lo = vget_low_s16(v_val_2);
- const int16x4_t v_val_3_lo = vget_low_s16(v_val_3);
- int32x4_t v_sum_01 = vmull_s16(v_val_0_lo, v_val_0_lo);
- v_sum_01 = vmlal_s16(v_sum_01, v_val_1_lo, v_val_1_lo);
- int32x4_t v_sum_23 = vmull_s16(v_val_2_lo, v_val_2_lo);
- v_sum_23 = vmlal_s16(v_sum_23, v_val_3_lo, v_val_3_lo);
-#if defined(__aarch64__)
- v_sum_01 = vmlal_high_s16(v_sum_01, v_val_0, v_val_0);
- v_sum_01 = vmlal_high_s16(v_sum_01, v_val_1, v_val_1);
- v_sum_23 = vmlal_high_s16(v_sum_23, v_val_2, v_val_2);
- v_sum_23 = vmlal_high_s16(v_sum_23, v_val_3, v_val_3);
- v_sum = vaddq_s32(v_sum, vpaddq_s32(v_sum_01, v_sum_23));
-#else
- const int16x4_t v_val_0_hi = vget_high_s16(v_val_0);
- const int16x4_t v_val_1_hi = vget_high_s16(v_val_1);
- const int16x4_t v_val_2_hi = vget_high_s16(v_val_2);
- const int16x4_t v_val_3_hi = vget_high_s16(v_val_3);
- v_sum_01 = vmlal_s16(v_sum_01, v_val_0_hi, v_val_0_hi);
- v_sum_01 = vmlal_s16(v_sum_01, v_val_1_hi, v_val_1_hi);
- v_sum_23 = vmlal_s16(v_sum_23, v_val_2_hi, v_val_2_hi);
- v_sum_23 = vmlal_s16(v_sum_23, v_val_3_hi, v_val_3_hi);
- v_sum = vaddq_s32(v_sum, vcombine_s32(vqmovn_s64(vpaddlq_s32(v_sum_01)),
- vqmovn_s64(vpaddlq_s32(v_sum_23))));
-#endif
- c += 8;
- } while (c < width);
-
- v_acc_q = vpadalq_u32(v_acc_q, vreinterpretq_u32_s32(v_sum));
+ sum_squares[0] = vmlal_s16(sum_squares[0], s0, s0);
+ sum_squares[0] = vmlal_s16(sum_squares[0], s1, s1);
+ sum_squares[1] = vmlal_s16(sum_squares[1], s2, s2);
+ sum_squares[1] = vmlal_s16(sum_squares[1], s3, s3);
src += 4 * stride;
- r += 4;
- } while (r < height);
-#if defined(__aarch64__)
- return vaddvq_u64(v_acc_q);
-#else
- v_acc_q = vaddq_u64(v_acc_q, vextq_u64(v_acc_q, v_acc_q, 1));
- return vgetq_lane_u64(v_acc_q, 0);
-#endif
+ h += 4;
+ } while (h < height);
+
+ return horizontal_long_add_u32x4(
+ vreinterpretq_u32_s32(vaddq_s32(sum_squares[0], sum_squares[1])));
+}
+
+static INLINE uint64_t aom_sum_squares_2d_i16_nxn_neon(const int16_t *src,
+ int stride, int width,
+ int height) {
+ uint64x2_t sum_squares = vdupq_n_u64(0);
+
+ int h = 0;
+ do {
+ int32x4_t ss_row[2] = { vdupq_n_s32(0), vdupq_n_s32(0) };
+ int w = 0;
+ do {
+ const int16_t *s = src + w;
+ int16x8_t s0 = vld1q_s16(s + 0 * stride);
+ int16x8_t s1 = vld1q_s16(s + 1 * stride);
+ int16x8_t s2 = vld1q_s16(s + 2 * stride);
+ int16x8_t s3 = vld1q_s16(s + 3 * stride);
+
+ ss_row[0] = vmlal_s16(ss_row[0], vget_low_s16(s0), vget_low_s16(s0));
+ ss_row[0] = vmlal_s16(ss_row[0], vget_low_s16(s1), vget_low_s16(s1));
+ ss_row[0] = vmlal_s16(ss_row[0], vget_low_s16(s2), vget_low_s16(s2));
+ ss_row[0] = vmlal_s16(ss_row[0], vget_low_s16(s3), vget_low_s16(s3));
+ ss_row[1] = vmlal_s16(ss_row[1], vget_high_s16(s0), vget_high_s16(s0));
+ ss_row[1] = vmlal_s16(ss_row[1], vget_high_s16(s1), vget_high_s16(s1));
+ ss_row[1] = vmlal_s16(ss_row[1], vget_high_s16(s2), vget_high_s16(s2));
+ ss_row[1] = vmlal_s16(ss_row[1], vget_high_s16(s3), vget_high_s16(s3));
+ w += 8;
+ } while (w < width);
+
+ sum_squares = vpadalq_u32(
+ sum_squares, vreinterpretq_u32_s32(vaddq_s32(ss_row[0], ss_row[1])));
+
+ src += 4 * stride;
+ h += 4;
+ } while (h < height);
+
+ return horizontal_add_u64x2(sum_squares);
}
uint64_t aom_sum_squares_2d_i16_neon(const int16_t *src, int stride, int width,
@@ -136,3 +108,118 @@
return aom_sum_squares_2d_i16_c(src, stride, width, height);
}
}
+
+static INLINE uint64_t aom_sum_sse_2d_i16_4x4_neon(const int16_t *src,
+ int stride, int *sum) {
+ int16x4_t s0 = vld1_s16(src + 0 * stride);
+ int16x4_t s1 = vld1_s16(src + 1 * stride);
+ int16x4_t s2 = vld1_s16(src + 2 * stride);
+ int16x4_t s3 = vld1_s16(src + 3 * stride);
+
+ int32x4_t sse = vmull_s16(s0, s0);
+ sse = vmlal_s16(sse, s1, s1);
+ sse = vmlal_s16(sse, s2, s2);
+ sse = vmlal_s16(sse, s3, s3);
+
+ int32x4_t sum_01 = vaddl_s16(s0, s1);
+ int32x4_t sum_23 = vaddl_s16(s2, s3);
+ *sum += horizontal_add_s32x4(vaddq_s32(sum_01, sum_23));
+
+ return horizontal_long_add_u32x4(vreinterpretq_u32_s32(sse));
+}
+
+static INLINE uint64_t aom_sum_sse_2d_i16_4xn_neon(const int16_t *src,
+ int stride, int height,
+ int *sum) {
+ int32x4_t sse[2] = { vdupq_n_s32(0), vdupq_n_s32(0) };
+ int32x2_t sum_acc[2] = { vdup_n_s32(0), vdup_n_s32(0) };
+
+ int h = 0;
+ do {
+ int16x4_t s0 = vld1_s16(src + 0 * stride);
+ int16x4_t s1 = vld1_s16(src + 1 * stride);
+ int16x4_t s2 = vld1_s16(src + 2 * stride);
+ int16x4_t s3 = vld1_s16(src + 3 * stride);
+
+ sse[0] = vmlal_s16(sse[0], s0, s0);
+ sse[0] = vmlal_s16(sse[0], s1, s1);
+ sse[1] = vmlal_s16(sse[1], s2, s2);
+ sse[1] = vmlal_s16(sse[1], s3, s3);
+
+ sum_acc[0] = vpadal_s16(sum_acc[0], s0);
+ sum_acc[0] = vpadal_s16(sum_acc[0], s1);
+ sum_acc[1] = vpadal_s16(sum_acc[1], s2);
+ sum_acc[1] = vpadal_s16(sum_acc[1], s3);
+
+ src += 4 * stride;
+ h += 4;
+ } while (h < height);
+
+ *sum += horizontal_add_s32x4(vcombine_s32(sum_acc[0], sum_acc[1]));
+ return horizontal_long_add_u32x4(
+ vreinterpretq_u32_s32(vaddq_s32(sse[0], sse[1])));
+}
+
+static INLINE uint64_t aom_sum_sse_2d_i16_nxn_neon(const int16_t *src,
+ int stride, int width,
+ int height, int *sum) {
+ uint64x2_t sse = vdupq_n_u64(0);
+ int32x4_t sum_acc = vdupq_n_s32(0);
+
+ int h = 0;
+ do {
+ int32x4_t sse_row[2] = { vdupq_n_s32(0), vdupq_n_s32(0) };
+ int w = 0;
+ do {
+ const int16_t *s = src + w;
+ int16x8_t s0 = vld1q_s16(s + 0 * stride);
+ int16x8_t s1 = vld1q_s16(s + 1 * stride);
+ int16x8_t s2 = vld1q_s16(s + 2 * stride);
+ int16x8_t s3 = vld1q_s16(s + 3 * stride);
+
+ sse_row[0] = vmlal_s16(sse_row[0], vget_low_s16(s0), vget_low_s16(s0));
+ sse_row[0] = vmlal_s16(sse_row[0], vget_low_s16(s1), vget_low_s16(s1));
+ sse_row[0] = vmlal_s16(sse_row[0], vget_low_s16(s2), vget_low_s16(s2));
+ sse_row[0] = vmlal_s16(sse_row[0], vget_low_s16(s3), vget_low_s16(s3));
+ sse_row[1] = vmlal_s16(sse_row[1], vget_high_s16(s0), vget_high_s16(s0));
+ sse_row[1] = vmlal_s16(sse_row[1], vget_high_s16(s1), vget_high_s16(s1));
+ sse_row[1] = vmlal_s16(sse_row[1], vget_high_s16(s2), vget_high_s16(s2));
+ sse_row[1] = vmlal_s16(sse_row[1], vget_high_s16(s3), vget_high_s16(s3));
+
+ sum_acc = vpadalq_s16(sum_acc, s0);
+ sum_acc = vpadalq_s16(sum_acc, s1);
+ sum_acc = vpadalq_s16(sum_acc, s2);
+ sum_acc = vpadalq_s16(sum_acc, s3);
+
+ w += 8;
+ } while (w < width);
+
+ sse = vpadalq_u32(sse,
+ vreinterpretq_u32_s32(vaddq_s32(sse_row[0], sse_row[1])));
+
+ src += 4 * stride;
+ h += 4;
+ } while (h < height);
+
+ *sum += horizontal_add_s32x4(sum_acc);
+ return horizontal_add_u64x2(sse);
+}
+
+uint64_t aom_sum_sse_2d_i16_neon(const int16_t *src, int stride, int width,
+ int height, int *sum) {
+ uint64_t sse;
+
+ if (LIKELY(width == 4 && height == 4)) {
+ sse = aom_sum_sse_2d_i16_4x4_neon(src, stride, sum);
+ } else if (LIKELY(width == 4 && (height & 3) == 0)) {
+ // width = 4, height is a multiple of 4.
+ sse = aom_sum_sse_2d_i16_4xn_neon(src, stride, height, sum);
+ } else if (LIKELY((width & 7) == 0 && (height & 3) == 0)) {
+ // Generic case - width is multiple of 8, height is multiple of 4.
+ sse = aom_sum_sse_2d_i16_nxn_neon(src, stride, width, height, sum);
+ } else {
+ sse = aom_sum_sse_2d_i16_c(src, stride, width, height, sum);
+ }
+
+ return sse;
+}
diff --git a/aom_dsp/arm/variance_neon.c b/aom_dsp/arm/variance_neon.c
index 3378491..f078705 100644
--- a/aom_dsp/arm/variance_neon.c
+++ b/aom_dsp/arm/variance_neon.c
@@ -13,548 +13,177 @@
#include "config/aom_dsp_rtcd.h"
#include "config/aom_config.h"
+#include "aom_dsp/arm/mem_neon.h"
#include "aom_dsp/arm/sum_neon.h"
#include "aom/aom_integer.h"
#include "aom_ports/mem.h"
-// w * h must be less than 2048 or local variable v_sum may overflow.
-static void variance_neon_w8(const uint8_t *a, int a_stride, const uint8_t *b,
- int b_stride, int w, int h, uint32_t *sse,
- int *sum) {
- int i, j;
- int16x8_t v_sum = vdupq_n_s16(0);
- int32x4_t v_sse_lo = vdupq_n_s32(0);
- int32x4_t v_sse_hi = vdupq_n_s32(0);
+#if defined(__ARM_FEATURE_DOTPROD)
- for (i = 0; i < h; ++i) {
- for (j = 0; j < w; j += 8) {
- const uint8x8_t v_a = vld1_u8(&a[j]);
- const uint8x8_t v_b = vld1_u8(&b[j]);
- const uint16x8_t v_diff = vsubl_u8(v_a, v_b);
- const int16x8_t sv_diff = vreinterpretq_s16_u16(v_diff);
- v_sum = vaddq_s16(v_sum, sv_diff);
- v_sse_lo =
- vmlal_s16(v_sse_lo, vget_low_s16(sv_diff), vget_low_s16(sv_diff));
- v_sse_hi =
- vmlal_s16(v_sse_hi, vget_high_s16(sv_diff), vget_high_s16(sv_diff));
- }
- a += a_stride;
- b += b_stride;
- }
-
- *sum = horizontal_add_s16x8(v_sum);
- *sse = (unsigned int)horizontal_add_s32x4(vaddq_s32(v_sse_lo, v_sse_hi));
-}
-
-void aom_get8x8var_neon(const uint8_t *a, int a_stride, const uint8_t *b,
- int b_stride, unsigned int *sse, int *sum) {
- variance_neon_w8(a, a_stride, b, b_stride, 8, 8, sse, sum);
-}
-
-void aom_get16x16var_neon(const uint8_t *a, int a_stride, const uint8_t *b,
- int b_stride, unsigned int *sse, int *sum) {
- variance_neon_w8(a, a_stride, b, b_stride, 16, 16, sse, sum);
-}
-
-// TODO(yunqingwang): Perform variance of two/four 8x8 blocks similar to that of
-// AVX2.
-void aom_get_sse_sum_8x8_quad_neon(const uint8_t *a, int a_stride,
- const uint8_t *b, int b_stride,
- unsigned int *sse, int *sum) {
- // Loop over 4 8x8 blocks. Process one 8x32 block.
- for (int k = 0; k < 4; k++) {
- variance_neon_w8(a + (k * 8), a_stride, b + (k * 8), b_stride, 8, 8,
- &sse[k], &sum[k]);
- }
-}
-
-unsigned int aom_variance8x8_neon(const uint8_t *a, int a_stride,
- const uint8_t *b, int b_stride,
- unsigned int *sse) {
- int sum;
- variance_neon_w8(a, a_stride, b, b_stride, 8, 8, sse, &sum);
- return *sse - ((sum * sum) >> 6);
-}
-
-unsigned int aom_variance16x16_neon(const uint8_t *a, int a_stride,
- const uint8_t *b, int b_stride,
- unsigned int *sse) {
- int sum;
- variance_neon_w8(a, a_stride, b, b_stride, 16, 16, sse, &sum);
- return *sse - (((unsigned int)((int64_t)sum * sum)) >> 8);
-}
-
-unsigned int aom_variance32x32_neon(const uint8_t *a, int a_stride,
- const uint8_t *b, int b_stride,
- unsigned int *sse) {
- int sum;
- variance_neon_w8(a, a_stride, b, b_stride, 32, 32, sse, &sum);
- return *sse - (unsigned int)(((int64_t)sum * sum) >> 10);
-}
-
-unsigned int aom_variance32x64_neon(const uint8_t *a, int a_stride,
- const uint8_t *b, int b_stride,
- unsigned int *sse) {
- int sum1, sum2;
- uint32_t sse1, sse2;
- variance_neon_w8(a, a_stride, b, b_stride, 32, 32, &sse1, &sum1);
- variance_neon_w8(a + (32 * a_stride), a_stride, b + (32 * b_stride), b_stride,
- 32, 32, &sse2, &sum2);
- *sse = sse1 + sse2;
- sum1 += sum2;
- return *sse - (unsigned int)(((int64_t)sum1 * sum1) >> 11);
-}
-
-unsigned int aom_variance64x32_neon(const uint8_t *a, int a_stride,
- const uint8_t *b, int b_stride,
- unsigned int *sse) {
- int sum1, sum2;
- uint32_t sse1, sse2;
- variance_neon_w8(a, a_stride, b, b_stride, 64, 16, &sse1, &sum1);
- variance_neon_w8(a + (16 * a_stride), a_stride, b + (16 * b_stride), b_stride,
- 64, 16, &sse2, &sum2);
- *sse = sse1 + sse2;
- sum1 += sum2;
- return *sse - (unsigned int)(((int64_t)sum1 * sum1) >> 11);
-}
-
-unsigned int aom_variance64x64_neon(const uint8_t *a, int a_stride,
- const uint8_t *b, int b_stride,
- unsigned int *sse) {
- int sum1, sum2;
- uint32_t sse1, sse2;
-
- variance_neon_w8(a, a_stride, b, b_stride, 64, 16, &sse1, &sum1);
- variance_neon_w8(a + (16 * a_stride), a_stride, b + (16 * b_stride), b_stride,
- 64, 16, &sse2, &sum2);
- sse1 += sse2;
- sum1 += sum2;
-
- variance_neon_w8(a + (16 * 2 * a_stride), a_stride, b + (16 * 2 * b_stride),
- b_stride, 64, 16, &sse2, &sum2);
- sse1 += sse2;
- sum1 += sum2;
-
- variance_neon_w8(a + (16 * 3 * a_stride), a_stride, b + (16 * 3 * b_stride),
- b_stride, 64, 16, &sse2, &sum2);
- *sse = sse1 + sse2;
- sum1 += sum2;
- return *sse - (unsigned int)(((int64_t)sum1 * sum1) >> 12);
-}
-
-unsigned int aom_variance128x128_neon(const uint8_t *a, int a_stride,
- const uint8_t *b, int b_stride,
- unsigned int *sse) {
- int sum1, sum2;
- uint32_t sse1, sse2;
- sum1 = sse1 = 0;
- for (int i = 0; i < 16; i++) {
- variance_neon_w8(a + (8 * i * a_stride), a_stride, b + (8 * i * b_stride),
- b_stride, 128, 8, &sse2, &sum2);
- sse1 += sse2;
- sum1 += sum2;
- }
-
- *sse = sse1;
-
- return *sse - (unsigned int)(((int64_t)sum1 * sum1) >> 14);
-}
-
-unsigned int aom_variance16x8_neon(const unsigned char *src_ptr,
- int source_stride,
- const unsigned char *ref_ptr,
- int recon_stride, unsigned int *sse) {
- int i;
- int16x4_t d22s16, d23s16, d24s16, d25s16, d26s16, d27s16, d28s16, d29s16;
- uint32x2_t d0u32, d10u32;
- int64x1_t d0s64, d1s64;
- uint8x16_t q0u8, q1u8, q2u8, q3u8;
- uint16x8_t q11u16, q12u16, q13u16, q14u16;
- int32x4_t q8s32, q9s32, q10s32;
- int64x2_t q0s64, q1s64, q5s64;
-
- q8s32 = vdupq_n_s32(0);
- q9s32 = vdupq_n_s32(0);
- q10s32 = vdupq_n_s32(0);
-
- for (i = 0; i < 4; i++) {
- q0u8 = vld1q_u8(src_ptr);
- src_ptr += source_stride;
- q1u8 = vld1q_u8(src_ptr);
- src_ptr += source_stride;
- __builtin_prefetch(src_ptr);
-
- q2u8 = vld1q_u8(ref_ptr);
- ref_ptr += recon_stride;
- q3u8 = vld1q_u8(ref_ptr);
- ref_ptr += recon_stride;
- __builtin_prefetch(ref_ptr);
-
- q11u16 = vsubl_u8(vget_low_u8(q0u8), vget_low_u8(q2u8));
- q12u16 = vsubl_u8(vget_high_u8(q0u8), vget_high_u8(q2u8));
- q13u16 = vsubl_u8(vget_low_u8(q1u8), vget_low_u8(q3u8));
- q14u16 = vsubl_u8(vget_high_u8(q1u8), vget_high_u8(q3u8));
-
- d22s16 = vreinterpret_s16_u16(vget_low_u16(q11u16));
- d23s16 = vreinterpret_s16_u16(vget_high_u16(q11u16));
- q8s32 = vpadalq_s16(q8s32, vreinterpretq_s16_u16(q11u16));
- q9s32 = vmlal_s16(q9s32, d22s16, d22s16);
- q10s32 = vmlal_s16(q10s32, d23s16, d23s16);
-
- d24s16 = vreinterpret_s16_u16(vget_low_u16(q12u16));
- d25s16 = vreinterpret_s16_u16(vget_high_u16(q12u16));
- q8s32 = vpadalq_s16(q8s32, vreinterpretq_s16_u16(q12u16));
- q9s32 = vmlal_s16(q9s32, d24s16, d24s16);
- q10s32 = vmlal_s16(q10s32, d25s16, d25s16);
-
- d26s16 = vreinterpret_s16_u16(vget_low_u16(q13u16));
- d27s16 = vreinterpret_s16_u16(vget_high_u16(q13u16));
- q8s32 = vpadalq_s16(q8s32, vreinterpretq_s16_u16(q13u16));
- q9s32 = vmlal_s16(q9s32, d26s16, d26s16);
- q10s32 = vmlal_s16(q10s32, d27s16, d27s16);
-
- d28s16 = vreinterpret_s16_u16(vget_low_u16(q14u16));
- d29s16 = vreinterpret_s16_u16(vget_high_u16(q14u16));
- q8s32 = vpadalq_s16(q8s32, vreinterpretq_s16_u16(q14u16));
- q9s32 = vmlal_s16(q9s32, d28s16, d28s16);
- q10s32 = vmlal_s16(q10s32, d29s16, d29s16);
- }
-
- q10s32 = vaddq_s32(q10s32, q9s32);
- q0s64 = vpaddlq_s32(q8s32);
- q1s64 = vpaddlq_s32(q10s32);
-
- d0s64 = vadd_s64(vget_low_s64(q0s64), vget_high_s64(q0s64));
- d1s64 = vadd_s64(vget_low_s64(q1s64), vget_high_s64(q1s64));
-
- q5s64 = vmull_s32(vreinterpret_s32_s64(d0s64), vreinterpret_s32_s64(d0s64));
- vst1_lane_u32((uint32_t *)sse, vreinterpret_u32_s64(d1s64), 0);
-
- d10u32 = vshr_n_u32(vreinterpret_u32_s64(vget_low_s64(q5s64)), 7);
- d0u32 = vsub_u32(vreinterpret_u32_s64(d1s64), d10u32);
-
- return vget_lane_u32(d0u32, 0);
-}
-
-unsigned int aom_variance8x16_neon(const unsigned char *src_ptr,
- int source_stride,
- const unsigned char *ref_ptr,
- int recon_stride, unsigned int *sse) {
- int i;
- uint8x8_t d0u8, d2u8, d4u8, d6u8;
- int16x4_t d22s16, d23s16, d24s16, d25s16;
- uint32x2_t d0u32, d10u32;
- int64x1_t d0s64, d1s64;
- uint16x8_t q11u16, q12u16;
- int32x4_t q8s32, q9s32, q10s32;
- int64x2_t q0s64, q1s64, q5s64;
-
- q8s32 = vdupq_n_s32(0);
- q9s32 = vdupq_n_s32(0);
- q10s32 = vdupq_n_s32(0);
-
- for (i = 0; i < 8; i++) {
- d0u8 = vld1_u8(src_ptr);
- src_ptr += source_stride;
- d2u8 = vld1_u8(src_ptr);
- src_ptr += source_stride;
- __builtin_prefetch(src_ptr);
-
- d4u8 = vld1_u8(ref_ptr);
- ref_ptr += recon_stride;
- d6u8 = vld1_u8(ref_ptr);
- ref_ptr += recon_stride;
- __builtin_prefetch(ref_ptr);
-
- q11u16 = vsubl_u8(d0u8, d4u8);
- q12u16 = vsubl_u8(d2u8, d6u8);
-
- d22s16 = vreinterpret_s16_u16(vget_low_u16(q11u16));
- d23s16 = vreinterpret_s16_u16(vget_high_u16(q11u16));
- q8s32 = vpadalq_s16(q8s32, vreinterpretq_s16_u16(q11u16));
- q9s32 = vmlal_s16(q9s32, d22s16, d22s16);
- q10s32 = vmlal_s16(q10s32, d23s16, d23s16);
-
- d24s16 = vreinterpret_s16_u16(vget_low_u16(q12u16));
- d25s16 = vreinterpret_s16_u16(vget_high_u16(q12u16));
- q8s32 = vpadalq_s16(q8s32, vreinterpretq_s16_u16(q12u16));
- q9s32 = vmlal_s16(q9s32, d24s16, d24s16);
- q10s32 = vmlal_s16(q10s32, d25s16, d25s16);
- }
-
- q10s32 = vaddq_s32(q10s32, q9s32);
- q0s64 = vpaddlq_s32(q8s32);
- q1s64 = vpaddlq_s32(q10s32);
-
- d0s64 = vadd_s64(vget_low_s64(q0s64), vget_high_s64(q0s64));
- d1s64 = vadd_s64(vget_low_s64(q1s64), vget_high_s64(q1s64));
-
- q5s64 = vmull_s32(vreinterpret_s32_s64(d0s64), vreinterpret_s32_s64(d0s64));
- vst1_lane_u32((uint32_t *)sse, vreinterpret_u32_s64(d1s64), 0);
-
- d10u32 = vshr_n_u32(vreinterpret_u32_s64(vget_low_s64(q5s64)), 7);
- d0u32 = vsub_u32(vreinterpret_u32_s64(d1s64), d10u32);
-
- return vget_lane_u32(d0u32, 0);
-}
-
-unsigned int aom_mse16x16_neon(const unsigned char *src_ptr, int source_stride,
- const unsigned char *ref_ptr, int recon_stride,
- unsigned int *sse) {
- int i;
- int16x4_t d22s16, d23s16, d24s16, d25s16, d26s16, d27s16, d28s16, d29s16;
- int64x1_t d0s64;
- uint8x16_t q0u8, q1u8, q2u8, q3u8;
- int32x4_t q7s32, q8s32, q9s32, q10s32;
- uint16x8_t q11u16, q12u16, q13u16, q14u16;
- int64x2_t q1s64;
-
- q7s32 = vdupq_n_s32(0);
- q8s32 = vdupq_n_s32(0);
- q9s32 = vdupq_n_s32(0);
- q10s32 = vdupq_n_s32(0);
-
- for (i = 0; i < 8; i++) { // mse16x16_neon_loop
- q0u8 = vld1q_u8(src_ptr);
- src_ptr += source_stride;
- q1u8 = vld1q_u8(src_ptr);
- src_ptr += source_stride;
- q2u8 = vld1q_u8(ref_ptr);
- ref_ptr += recon_stride;
- q3u8 = vld1q_u8(ref_ptr);
- ref_ptr += recon_stride;
-
- q11u16 = vsubl_u8(vget_low_u8(q0u8), vget_low_u8(q2u8));
- q12u16 = vsubl_u8(vget_high_u8(q0u8), vget_high_u8(q2u8));
- q13u16 = vsubl_u8(vget_low_u8(q1u8), vget_low_u8(q3u8));
- q14u16 = vsubl_u8(vget_high_u8(q1u8), vget_high_u8(q3u8));
-
- d22s16 = vreinterpret_s16_u16(vget_low_u16(q11u16));
- d23s16 = vreinterpret_s16_u16(vget_high_u16(q11u16));
- q7s32 = vmlal_s16(q7s32, d22s16, d22s16);
- q8s32 = vmlal_s16(q8s32, d23s16, d23s16);
-
- d24s16 = vreinterpret_s16_u16(vget_low_u16(q12u16));
- d25s16 = vreinterpret_s16_u16(vget_high_u16(q12u16));
- q9s32 = vmlal_s16(q9s32, d24s16, d24s16);
- q10s32 = vmlal_s16(q10s32, d25s16, d25s16);
-
- d26s16 = vreinterpret_s16_u16(vget_low_u16(q13u16));
- d27s16 = vreinterpret_s16_u16(vget_high_u16(q13u16));
- q7s32 = vmlal_s16(q7s32, d26s16, d26s16);
- q8s32 = vmlal_s16(q8s32, d27s16, d27s16);
-
- d28s16 = vreinterpret_s16_u16(vget_low_u16(q14u16));
- d29s16 = vreinterpret_s16_u16(vget_high_u16(q14u16));
- q9s32 = vmlal_s16(q9s32, d28s16, d28s16);
- q10s32 = vmlal_s16(q10s32, d29s16, d29s16);
- }
-
- q7s32 = vaddq_s32(q7s32, q8s32);
- q9s32 = vaddq_s32(q9s32, q10s32);
- q10s32 = vaddq_s32(q7s32, q9s32);
-
- q1s64 = vpaddlq_s32(q10s32);
- d0s64 = vadd_s64(vget_low_s64(q1s64), vget_high_s64(q1s64));
-
- vst1_lane_u32((uint32_t *)sse, vreinterpret_u32_s64(d0s64), 0);
- return vget_lane_u32(vreinterpret_u32_s64(d0s64), 0);
-}
-
-unsigned int aom_get4x4sse_cs_neon(const unsigned char *src_ptr,
- int source_stride,
- const unsigned char *ref_ptr,
- int recon_stride) {
- int16x4_t d22s16, d24s16, d26s16, d28s16;
- int64x1_t d0s64;
- uint8x8_t d0u8, d1u8, d2u8, d3u8, d4u8, d5u8, d6u8, d7u8;
- int32x4_t q7s32, q8s32, q9s32, q10s32;
- uint16x8_t q11u16, q12u16, q13u16, q14u16;
- int64x2_t q1s64;
-
- d0u8 = vld1_u8(src_ptr);
- src_ptr += source_stride;
- d4u8 = vld1_u8(ref_ptr);
- ref_ptr += recon_stride;
- d1u8 = vld1_u8(src_ptr);
- src_ptr += source_stride;
- d5u8 = vld1_u8(ref_ptr);
- ref_ptr += recon_stride;
- d2u8 = vld1_u8(src_ptr);
- src_ptr += source_stride;
- d6u8 = vld1_u8(ref_ptr);
- ref_ptr += recon_stride;
- d3u8 = vld1_u8(src_ptr);
- d7u8 = vld1_u8(ref_ptr);
-
- q11u16 = vsubl_u8(d0u8, d4u8);
- q12u16 = vsubl_u8(d1u8, d5u8);
- q13u16 = vsubl_u8(d2u8, d6u8);
- q14u16 = vsubl_u8(d3u8, d7u8);
-
- d22s16 = vget_low_s16(vreinterpretq_s16_u16(q11u16));
- d24s16 = vget_low_s16(vreinterpretq_s16_u16(q12u16));
- d26s16 = vget_low_s16(vreinterpretq_s16_u16(q13u16));
- d28s16 = vget_low_s16(vreinterpretq_s16_u16(q14u16));
-
- q7s32 = vmull_s16(d22s16, d22s16);
- q8s32 = vmull_s16(d24s16, d24s16);
- q9s32 = vmull_s16(d26s16, d26s16);
- q10s32 = vmull_s16(d28s16, d28s16);
-
- q7s32 = vaddq_s32(q7s32, q8s32);
- q9s32 = vaddq_s32(q9s32, q10s32);
- q9s32 = vaddq_s32(q7s32, q9s32);
-
- q1s64 = vpaddlq_s32(q9s32);
- d0s64 = vadd_s64(vget_low_s64(q1s64), vget_high_s64(q1s64));
-
- return vget_lane_u32(vreinterpret_u32_s64(d0s64), 0);
-}
-
-// Load 4 sets of 4 bytes when alignment is not guaranteed.
-static INLINE uint8x16_t load_unaligned_u8q(const uint8_t *buf, int stride) {
- uint32_t a;
- uint32x4_t a_u32 = vdupq_n_u32(0);
- if (stride == 4) return vld1q_u8(buf);
- memcpy(&a, buf, 4);
- buf += stride;
- a_u32 = vld1q_lane_u32(&a, a_u32, 0);
- memcpy(&a, buf, 4);
- buf += stride;
- a_u32 = vld1q_lane_u32(&a, a_u32, 1);
- memcpy(&a, buf, 4);
- buf += stride;
- a_u32 = vld1q_lane_u32(&a, a_u32, 2);
- memcpy(&a, buf, 4);
- buf += stride;
- a_u32 = vld1q_lane_u32(&a, a_u32, 3);
- return vreinterpretq_u8_u32(a_u32);
-}
-
-// The variance helper functions use int16_t for sum. 8 values are accumulated
-// and then added (at which point they expand up to int32_t). To avoid overflow,
-// there can be no more than 32767 / 255 ~= 128 values accumulated in each
-// column. For a 32x32 buffer, this results in 32 / 8 = 4 values per row * 32
-// rows = 128. Asserts have been added to each function to warn against reaching
-// this limit.
-
-// Process a block of width 4 four rows at a time.
-static void variance_neon_w4x4(const uint8_t *a, int a_stride, const uint8_t *b,
- int b_stride, int h, uint32_t *sse, int *sum) {
- const int32x4_t zero = vdupq_n_s32(0);
- int16x8_t sum_s16 = vreinterpretq_s16_s32(zero);
- int32x4_t sse_s32 = zero;
-
- // Since width is only 4, sum_s16 only loads a half row per loop.
- assert(h <= 256);
-
- int i;
- for (i = 0; i < h; i += 4) {
- const uint8x16_t a_u8 = load_unaligned_u8q(a, a_stride);
- const uint8x16_t b_u8 = load_unaligned_u8q(b, b_stride);
- const int16x8_t diff_lo_s16 =
- vreinterpretq_s16_u16(vsubl_u8(vget_low_u8(a_u8), vget_low_u8(b_u8)));
- const int16x8_t diff_hi_s16 =
- vreinterpretq_s16_u16(vsubl_u8(vget_high_u8(a_u8), vget_high_u8(b_u8)));
-
- sum_s16 = vaddq_s16(sum_s16, diff_lo_s16);
- sum_s16 = vaddq_s16(sum_s16, diff_hi_s16);
-
- sse_s32 = vmlal_s16(sse_s32, vget_low_s16(diff_lo_s16),
- vget_low_s16(diff_lo_s16));
- sse_s32 = vmlal_s16(sse_s32, vget_high_s16(diff_lo_s16),
- vget_high_s16(diff_lo_s16));
-
- sse_s32 = vmlal_s16(sse_s32, vget_low_s16(diff_hi_s16),
- vget_low_s16(diff_hi_s16));
- sse_s32 = vmlal_s16(sse_s32, vget_high_s16(diff_hi_s16),
- vget_high_s16(diff_hi_s16));
-
- a += 4 * a_stride;
- b += 4 * b_stride;
- }
-
- *sum = horizontal_add_s16x8(sum_s16);
- *sse = (uint32_t)horizontal_add_s32x4(sse_s32);
-}
-
-// Process a block of any size where the width is divisible by 16.
-static void variance_neon_w16(const uint8_t *a, int a_stride, const uint8_t *b,
- int b_stride, int w, int h, uint32_t *sse,
- int *sum) {
- const int32x4_t zero = vdupq_n_s32(0);
- int16x8_t sum_s16 = vreinterpretq_s16_s32(zero);
- int32x4_t sse_s32 = zero;
-
- // The loop loads 16 values at a time but doubles them up when accumulating
- // into sum_s16.
- assert(w / 8 * h <= 128);
-
- int i, j;
- for (i = 0; i < h; ++i) {
- for (j = 0; j < w; j += 16) {
- const uint8x16_t a_u8 = vld1q_u8(a + j);
- const uint8x16_t b_u8 = vld1q_u8(b + j);
-
- const int16x8_t diff_lo_s16 =
- vreinterpretq_s16_u16(vsubl_u8(vget_low_u8(a_u8), vget_low_u8(b_u8)));
- const int16x8_t diff_hi_s16 = vreinterpretq_s16_u16(
- vsubl_u8(vget_high_u8(a_u8), vget_high_u8(b_u8)));
-
- sum_s16 = vaddq_s16(sum_s16, diff_lo_s16);
- sum_s16 = vaddq_s16(sum_s16, diff_hi_s16);
-
- sse_s32 = vmlal_s16(sse_s32, vget_low_s16(diff_lo_s16),
- vget_low_s16(diff_lo_s16));
- sse_s32 = vmlal_s16(sse_s32, vget_high_s16(diff_lo_s16),
- vget_high_s16(diff_lo_s16));
-
- sse_s32 = vmlal_s16(sse_s32, vget_low_s16(diff_hi_s16),
- vget_low_s16(diff_hi_s16));
- sse_s32 = vmlal_s16(sse_s32, vget_high_s16(diff_hi_s16),
- vget_high_s16(diff_hi_s16));
- }
- a += a_stride;
- b += b_stride;
- }
-
- *sum = horizontal_add_s16x8(sum_s16);
- *sse = (uint32_t)horizontal_add_s32x4(sse_s32);
-}
-
-// Process a block of width 8 two rows at a time.
-static void variance_neon_w8x2(const uint8_t *a, int a_stride, const uint8_t *b,
- int b_stride, int h, uint32_t *sse, int *sum) {
- const int32x4_t zero = vdupq_n_s32(0);
- int16x8_t sum_s16 = vreinterpretq_s16_s32(zero);
- int32x4_t sse_s32 = zero;
-
- // Each column has it's own accumulator entry in sum_s16.
- assert(h <= 128);
+static INLINE void variance_4xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride, int h,
+ uint32_t *sse, int *sum) {
+ uint32x4_t src_sum = vdupq_n_u32(0);
+ uint32x4_t ref_sum = vdupq_n_u32(0);
+ uint32x4_t sse_u32 = vdupq_n_u32(0);
int i = 0;
do {
- const uint8x8_t a_0_u8 = vld1_u8(a);
- const uint8x8_t a_1_u8 = vld1_u8(a + a_stride);
- const uint8x8_t b_0_u8 = vld1_u8(b);
- const uint8x8_t b_1_u8 = vld1_u8(b + b_stride);
- const int16x8_t diff_0_s16 =
- vreinterpretq_s16_u16(vsubl_u8(a_0_u8, b_0_u8));
- const int16x8_t diff_1_s16 =
- vreinterpretq_s16_u16(vsubl_u8(a_1_u8, b_1_u8));
- sum_s16 = vaddq_s16(sum_s16, diff_0_s16);
- sum_s16 = vaddq_s16(sum_s16, diff_1_s16);
- sse_s32 =
- vmlal_s16(sse_s32, vget_low_s16(diff_0_s16), vget_low_s16(diff_0_s16));
- sse_s32 =
- vmlal_s16(sse_s32, vget_low_s16(diff_1_s16), vget_low_s16(diff_1_s16));
- sse_s32 = vmlal_s16(sse_s32, vget_high_s16(diff_0_s16),
- vget_high_s16(diff_0_s16));
- sse_s32 = vmlal_s16(sse_s32, vget_high_s16(diff_1_s16),
- vget_high_s16(diff_1_s16));
- a += a_stride + a_stride;
- b += b_stride + b_stride;
+ uint8x16_t s = load_unaligned_u8q(src, src_stride);
+ uint8x16_t r = load_unaligned_u8q(ref, ref_stride);
+
+ src_sum = vdotq_u32(src_sum, s, vdupq_n_u8(1));
+ ref_sum = vdotq_u32(ref_sum, r, vdupq_n_u8(1));
+
+ uint8x16_t abs_diff = vabdq_u8(s, r);
+ sse_u32 = vdotq_u32(sse_u32, abs_diff, abs_diff);
+
+ src += 4 * src_stride;
+ ref += 4 * ref_stride;
+ i += 4;
+ } while (i < h);
+
+ int32x4_t sum_diff =
+ vsubq_s32(vreinterpretq_s32_u32(src_sum), vreinterpretq_s32_u32(ref_sum));
+ *sum = horizontal_add_s32x4(sum_diff);
+ *sse = horizontal_add_u32x4(sse_u32);
+}
+
+static INLINE void variance_8xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride, int h,
+ uint32_t *sse, int *sum) {
+ uint32x4_t src_sum = vdupq_n_u32(0);
+ uint32x4_t ref_sum = vdupq_n_u32(0);
+ uint32x4_t sse_u32 = vdupq_n_u32(0);
+
+ int i = 0;
+ do {
+ uint8x16_t s = vcombine_u8(vld1_u8(src), vld1_u8(src + src_stride));
+ uint8x16_t r = vcombine_u8(vld1_u8(ref), vld1_u8(ref + ref_stride));
+
+ src_sum = vdotq_u32(src_sum, s, vdupq_n_u8(1));
+ ref_sum = vdotq_u32(ref_sum, r, vdupq_n_u8(1));
+
+ uint8x16_t abs_diff = vabdq_u8(s, r);
+ sse_u32 = vdotq_u32(sse_u32, abs_diff, abs_diff);
+
+ src += 2 * src_stride;
+ ref += 2 * ref_stride;
+ i += 2;
+ } while (i < h);
+
+ int32x4_t sum_diff =
+ vsubq_s32(vreinterpretq_s32_u32(src_sum), vreinterpretq_s32_u32(ref_sum));
+ *sum = horizontal_add_s32x4(sum_diff);
+ *sse = horizontal_add_u32x4(sse_u32);
+}
+
+static INLINE void variance_16xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride, int h,
+ uint32_t *sse, int *sum) {
+ uint32x4_t src_sum = vdupq_n_u32(0);
+ uint32x4_t ref_sum = vdupq_n_u32(0);
+ uint32x4_t sse_u32 = vdupq_n_u32(0);
+
+ int i = 0;
+ do {
+ uint8x16_t s = vld1q_u8(src);
+ uint8x16_t r = vld1q_u8(ref);
+
+ src_sum = vdotq_u32(src_sum, s, vdupq_n_u8(1));
+ ref_sum = vdotq_u32(ref_sum, r, vdupq_n_u8(1));
+
+ uint8x16_t abs_diff = vabdq_u8(s, r);
+ sse_u32 = vdotq_u32(sse_u32, abs_diff, abs_diff);
+
+ src += src_stride;
+ ref += ref_stride;
+ i++;
+ } while (i < h);
+
+ int32x4_t sum_diff =
+ vsubq_s32(vreinterpretq_s32_u32(src_sum), vreinterpretq_s32_u32(ref_sum));
+ *sum = horizontal_add_s32x4(sum_diff);
+ *sse = horizontal_add_u32x4(sse_u32);
+}
+
+static INLINE void variance_large_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ int w, int h, uint32_t *sse, int *sum) {
+ uint32x4_t src_sum = vdupq_n_u32(0);
+ uint32x4_t ref_sum = vdupq_n_u32(0);
+ uint32x4_t sse_u32 = vdupq_n_u32(0);
+
+ int i = 0;
+ do {
+ int j = 0;
+ do {
+ uint8x16_t s = vld1q_u8(src + j);
+ uint8x16_t r = vld1q_u8(ref + j);
+
+ src_sum = vdotq_u32(src_sum, s, vdupq_n_u8(1));
+ ref_sum = vdotq_u32(ref_sum, r, vdupq_n_u8(1));
+
+ uint8x16_t abs_diff = vabdq_u8(s, r);
+ sse_u32 = vdotq_u32(sse_u32, abs_diff, abs_diff);
+
+ j += 16;
+ } while (j < w);
+
+ src += src_stride;
+ ref += ref_stride;
+ i++;
+ } while (i < h);
+
+ int32x4_t sum_diff =
+ vsubq_s32(vreinterpretq_s32_u32(src_sum), vreinterpretq_s32_u32(ref_sum));
+ *sum = horizontal_add_s32x4(sum_diff);
+ *sse = horizontal_add_u32x4(sse_u32);
+}
+
+static INLINE void variance_32xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride, int h,
+ uint32_t *sse, int *sum) {
+ variance_large_neon(src, src_stride, ref, ref_stride, 32, h, sse, sum);
+}
+
+static INLINE void variance_64xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride, int h,
+ uint32_t *sse, int *sum) {
+ variance_large_neon(src, src_stride, ref, ref_stride, 64, h, sse, sum);
+}
+
+static INLINE void variance_128xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ int h, uint32_t *sse, int *sum) {
+ variance_large_neon(src, src_stride, ref, ref_stride, 128, h, sse, sum);
+}
+
+#else // !defined(__ARM_FEATURE_DOTPROD)
+
+static INLINE void variance_4xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride, int h,
+ uint32_t *sse, int *sum) {
+ int16x8_t sum_s16 = vdupq_n_s16(0);
+ int32x4_t sse_s32 = vdupq_n_s32(0);
+
+ // Number of rows we can process before 'sum_s16' overflows:
+ // 32767 / 255 ~= 128, but we use an 8-wide accumulator; so 256 4-wide rows.
+ assert(h <= 256);
+
+ int i = 0;
+ do {
+ uint8x8_t s = load_unaligned_u8(src, src_stride);
+ uint8x8_t r = load_unaligned_u8(ref, ref_stride);
+ int16x8_t diff = vreinterpretq_s16_u16(vsubl_u8(s, r));
+
+ sum_s16 = vaddq_s16(sum_s16, diff);
+
+ sse_s32 = vmlal_s16(sse_s32, vget_low_s16(diff), vget_low_s16(diff));
+ sse_s32 = vmlal_s16(sse_s32, vget_high_s16(diff), vget_high_s16(diff));
+
+ src += 2 * src_stride;
+ ref += 2 * ref_stride;
i += 2;
} while (i < h);
@@ -562,90 +191,426 @@
*sse = (uint32_t)horizontal_add_s32x4(sse_s32);
}
-#define VARIANCE_NXM(n, m, shift) \
- unsigned int aom_variance##n##x##m##_neon(const uint8_t *a, int a_stride, \
- const uint8_t *b, int b_stride, \
- unsigned int *sse) { \
- int sum; \
- if (n == 4) \
- variance_neon_w4x4(a, a_stride, b, b_stride, m, sse, &sum); \
- else if (n == 8) \
- variance_neon_w8x2(a, a_stride, b, b_stride, m, sse, &sum); \
- else \
- variance_neon_w16(a, a_stride, b, b_stride, n, m, sse, &sum); \
- if (n * m < 16 * 16) \
- return *sse - ((sum * sum) >> shift); \
- else \
- return *sse - (uint32_t)(((int64_t)sum * sum) >> shift); \
- }
+static INLINE void variance_8xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride, int h,
+ uint32_t *sse, int *sum) {
+ int16x8_t sum_s16 = vdupq_n_s16(0);
+ int32x4_t sse_s32[2] = { vdupq_n_s32(0), vdupq_n_s32(0) };
-static void variance_neon_wide_block(const uint8_t *a, int a_stride,
- const uint8_t *b, int b_stride, int w,
- int h, uint32_t *sse, int *sum) {
- const int32x4_t zero = vdupq_n_s32(0);
- int32x4_t v_diff = zero;
- int64x2_t v_sse = vreinterpretq_s64_s32(zero);
+ // Number of rows we can process before 'sum_s16' overflows:
+ // 32767 / 255 ~= 128
+ assert(h <= 128);
- int s, i, j;
- for (s = 0; s < 16; s++) {
- int32x4_t sse_s32 = zero;
- int16x8_t sum_s16 = vreinterpretq_s16_s32(zero);
- for (i = (s * h) >> 4; i < (((s + 1) * h) >> 4); ++i) {
- for (j = 0; j < w; j += 16) {
- const uint8x16_t a_u8 = vld1q_u8(a + j);
- const uint8x16_t b_u8 = vld1q_u8(b + j);
+ int i = 0;
+ do {
+ uint8x8_t s = vld1_u8(src);
+ uint8x8_t r = vld1_u8(ref);
+ int16x8_t diff = vreinterpretq_s16_u16(vsubl_u8(s, r));
- const int16x8_t diff_lo_s16 = vreinterpretq_s16_u16(
- vsubl_u8(vget_low_u8(a_u8), vget_low_u8(b_u8)));
- const int16x8_t diff_hi_s16 = vreinterpretq_s16_u16(
- vsubl_u8(vget_high_u8(a_u8), vget_high_u8(b_u8)));
+ sum_s16 = vaddq_s16(sum_s16, diff);
- sum_s16 = vaddq_s16(sum_s16, diff_lo_s16);
- sum_s16 = vaddq_s16(sum_s16, diff_hi_s16);
+ sse_s32[0] = vmlal_s16(sse_s32[0], vget_low_s16(diff), vget_low_s16(diff));
+ sse_s32[1] =
+ vmlal_s16(sse_s32[1], vget_high_s16(diff), vget_high_s16(diff));
- sse_s32 = vmlal_s16(sse_s32, vget_low_s16(diff_lo_s16),
- vget_low_s16(diff_lo_s16));
- sse_s32 = vmlal_s16(sse_s32, vget_high_s16(diff_lo_s16),
- vget_high_s16(diff_lo_s16));
- sse_s32 = vmlal_s16(sse_s32, vget_low_s16(diff_hi_s16),
- vget_low_s16(diff_hi_s16));
- sse_s32 = vmlal_s16(sse_s32, vget_high_s16(diff_hi_s16),
- vget_high_s16(diff_hi_s16));
- }
+ src += src_stride;
+ ref += ref_stride;
+ i++;
+ } while (i < h);
- a += a_stride;
- b += b_stride;
- }
-
- v_diff = vpadalq_s16(v_diff, sum_s16);
- v_sse = vpadalq_s32(v_sse, sse_s32);
- }
- int diff = horizontal_add_s32x4(v_diff);
-#if defined(__aarch64__)
- uint32_t sq = (uint32_t)vaddvq_u64(vreinterpretq_u64_s64(v_sse));
-#else
- uint32_t sq = vget_lane_u32(
- vreinterpret_u32_s64(vadd_s64(vget_low_s64(v_sse), vget_high_s64(v_sse))),
- 0);
-#endif
-
- *sum = diff;
- *sse = sq;
+ *sum = horizontal_add_s16x8(sum_s16);
+ *sse = (uint32_t)horizontal_add_s32x4(vaddq_s32(sse_s32[0], sse_s32[1]));
}
-#define VARIANCE_NXM_WIDE(W, H) \
- unsigned int aom_variance##W##x##H##_neon(const uint8_t *a, int a_stride, \
- const uint8_t *b, int b_stride, \
- uint32_t *sse) { \
- int sum; \
- variance_neon_wide_block(a, a_stride, b, b_stride, W, H, sse, &sum); \
- return *sse - (uint32_t)(((int64_t)sum * sum) / (W * H)); \
+static INLINE void variance_16xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride, int h,
+ uint32_t *sse, int *sum) {
+ int16x8_t sum_s16[2] = { vdupq_n_s16(0), vdupq_n_s16(0) };
+ int32x4_t sse_s32[2] = { vdupq_n_s32(0), vdupq_n_s32(0) };
+
+ // Number of rows we can process before 'sum_s16' accumulators overflow:
+ // 32767 / 255 ~= 128, so 128 16-wide rows.
+ assert(h <= 128);
+
+ int i = 0;
+ do {
+ uint8x16_t s = vld1q_u8(src);
+ uint8x16_t r = vld1q_u8(ref);
+
+ int16x8_t diff_l =
+ vreinterpretq_s16_u16(vsubl_u8(vget_low_u8(s), vget_low_u8(r)));
+ int16x8_t diff_h =
+ vreinterpretq_s16_u16(vsubl_u8(vget_high_u8(s), vget_high_u8(r)));
+
+ sum_s16[0] = vaddq_s16(sum_s16[0], diff_l);
+ sum_s16[1] = vaddq_s16(sum_s16[1], diff_h);
+
+ sse_s32[0] =
+ vmlal_s16(sse_s32[0], vget_low_s16(diff_l), vget_low_s16(diff_l));
+ sse_s32[1] =
+ vmlal_s16(sse_s32[1], vget_high_s16(diff_l), vget_high_s16(diff_l));
+ sse_s32[0] =
+ vmlal_s16(sse_s32[0], vget_low_s16(diff_h), vget_low_s16(diff_h));
+ sse_s32[1] =
+ vmlal_s16(sse_s32[1], vget_high_s16(diff_h), vget_high_s16(diff_h));
+
+ src += src_stride;
+ ref += ref_stride;
+ i++;
+ } while (i < h);
+
+ *sum = horizontal_add_s16x8(vaddq_s16(sum_s16[0], sum_s16[1]));
+ *sse = (uint32_t)horizontal_add_s32x4(vaddq_s32(sse_s32[0], sse_s32[1]));
+}
+
+static INLINE void variance_large_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ int w, int h, int h_limit, uint32_t *sse,
+ int *sum) {
+ int32x4_t sum_s32 = vdupq_n_s32(0);
+ int32x4_t sse_s32[2] = { vdupq_n_s32(0), vdupq_n_s32(0) };
+
+ // 'h_limit' is the number of 'w'-width rows we can process before our 16-bit
+ // accumulator overflows. After hitting this limit we accumulate into 32-bit
+ // elements.
+ int h_tmp = h > h_limit ? h_limit : h;
+
+ int i = 0;
+ do {
+ int16x8_t sum_s16[2] = { vdupq_n_s16(0), vdupq_n_s16(0) };
+ do {
+ int j = 0;
+ do {
+ uint8x16_t s = vld1q_u8(src + j);
+ uint8x16_t r = vld1q_u8(ref + j);
+
+ int16x8_t diff_l =
+ vreinterpretq_s16_u16(vsubl_u8(vget_low_u8(s), vget_low_u8(r)));
+ int16x8_t diff_h =
+ vreinterpretq_s16_u16(vsubl_u8(vget_high_u8(s), vget_high_u8(r)));
+
+ sum_s16[0] = vaddq_s16(sum_s16[0], diff_l);
+ sum_s16[1] = vaddq_s16(sum_s16[1], diff_h);
+
+ sse_s32[0] =
+ vmlal_s16(sse_s32[0], vget_low_s16(diff_l), vget_low_s16(diff_l));
+ sse_s32[1] =
+ vmlal_s16(sse_s32[1], vget_high_s16(diff_l), vget_high_s16(diff_l));
+ sse_s32[0] =
+ vmlal_s16(sse_s32[0], vget_low_s16(diff_h), vget_low_s16(diff_h));
+ sse_s32[1] =
+ vmlal_s16(sse_s32[1], vget_high_s16(diff_h), vget_high_s16(diff_h));
+
+ j += 16;
+ } while (j < w);
+
+ src += src_stride;
+ ref += ref_stride;
+ i++;
+ } while (i < h_tmp);
+
+ sum_s32 = vpadalq_s16(sum_s32, sum_s16[0]);
+ sum_s32 = vpadalq_s16(sum_s32, sum_s16[1]);
+
+ h_tmp += h_limit;
+ } while (i < h);
+
+ *sum = horizontal_add_s32x4(sum_s32);
+ *sse = (uint32_t)horizontal_add_s32x4(vaddq_s32(sse_s32[0], sse_s32[1]));
+}
+
+static INLINE void variance_32xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride, int h,
+ uint32_t *sse, int *sum) {
+ variance_large_neon(src, src_stride, ref, ref_stride, 32, h, 64, sse, sum);
+}
+
+static INLINE void variance_64xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride, int h,
+ uint32_t *sse, int *sum) {
+ variance_large_neon(src, src_stride, ref, ref_stride, 64, h, 32, sse, sum);
+}
+
+static INLINE void variance_128xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ int h, uint32_t *sse, int *sum) {
+ variance_large_neon(src, src_stride, ref, ref_stride, 128, h, 16, sse, sum);
+}
+
+#endif // defined(__ARM_FEATURE_DOTPROD)
+
+#define VARIANCE_WXH_NEON(w, h, shift) \
+ unsigned int aom_variance##w##x##h##_neon( \
+ const uint8_t *src, int src_stride, const uint8_t *ref, int ref_stride, \
+ unsigned int *sse) { \
+ int sum; \
+ variance_##w##xh_neon(src, src_stride, ref, ref_stride, h, sse, &sum); \
+ return *sse - (uint32_t)(((int64_t)sum * sum) >> shift); \
}
-VARIANCE_NXM(4, 4, 4)
-VARIANCE_NXM(4, 8, 5)
-VARIANCE_NXM(8, 4, 5)
-VARIANCE_NXM(16, 32, 9)
-VARIANCE_NXM(32, 16, 9)
-VARIANCE_NXM_WIDE(128, 64)
-VARIANCE_NXM_WIDE(64, 128)
+VARIANCE_WXH_NEON(4, 4, 4)
+VARIANCE_WXH_NEON(4, 8, 5)
+VARIANCE_WXH_NEON(4, 16, 6)
+
+VARIANCE_WXH_NEON(8, 4, 5)
+VARIANCE_WXH_NEON(8, 8, 6)
+VARIANCE_WXH_NEON(8, 16, 7)
+VARIANCE_WXH_NEON(8, 32, 8)
+
+VARIANCE_WXH_NEON(16, 4, 6)
+VARIANCE_WXH_NEON(16, 8, 7)
+VARIANCE_WXH_NEON(16, 16, 8)
+VARIANCE_WXH_NEON(16, 32, 9)
+VARIANCE_WXH_NEON(16, 64, 10)
+
+VARIANCE_WXH_NEON(32, 8, 8)
+VARIANCE_WXH_NEON(32, 16, 9)
+VARIANCE_WXH_NEON(32, 32, 10)
+VARIANCE_WXH_NEON(32, 64, 11)
+
+VARIANCE_WXH_NEON(64, 16, 10)
+VARIANCE_WXH_NEON(64, 32, 11)
+VARIANCE_WXH_NEON(64, 64, 12)
+VARIANCE_WXH_NEON(64, 128, 13)
+
+VARIANCE_WXH_NEON(128, 64, 13)
+VARIANCE_WXH_NEON(128, 128, 14)
+
+#undef VARIANCE_WXH_NEON
+
+void aom_get8x8var_neon(const uint8_t *src, int src_stride, const uint8_t *ref,
+ int ref_stride, unsigned int *sse, int *sum) {
+ variance_8xh_neon(src, src_stride, ref, ref_stride, 8, sse, sum);
+}
+
+void aom_get16x16var_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride, unsigned int *sse,
+ int *sum) {
+ variance_16xh_neon(src, src_stride, ref, ref_stride, 16, sse, sum);
+}
+
+// TODO(yunqingwang): Perform variance of two/four 8x8 blocks similar to that of
+// AVX2.
+void aom_get_sse_sum_8x8_quad_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ unsigned int *sse, int *sum) {
+ // Loop over 4 8x8 blocks. Process one 8x32 block.
+ for (int k = 0; k < 4; k++) {
+ variance_8xh_neon(src + (k * 8), src_stride, ref + (k * 8), ref_stride, 8,
+ &sse[k], &sum[k]);
+ }
+}
+
+#if defined(__ARM_FEATURE_DOTPROD)
+
+static INLINE unsigned int mse8xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ unsigned int *sse, int h) {
+ uint32x4_t sse_u32 = vdupq_n_u32(0);
+
+ int i = 0;
+ do {
+ uint8x16_t s = vcombine_u8(vld1_u8(src), vld1_u8(src + src_stride));
+ uint8x16_t r = vcombine_u8(vld1_u8(ref), vld1_u8(ref + ref_stride));
+
+ uint8x16_t abs_diff = vabdq_u8(s, r);
+
+ sse_u32 = vdotq_u32(sse_u32, abs_diff, abs_diff);
+
+ src += 2 * src_stride;
+ ref += 2 * ref_stride;
+ i += 2;
+ } while (i < h);
+
+ *sse = horizontal_add_u32x4(sse_u32);
+ return horizontal_add_u32x4(sse_u32);
+}
+
+static INLINE unsigned int mse16xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ unsigned int *sse, int h) {
+ uint32x4_t sse_u32[2] = { vdupq_n_u32(0), vdupq_n_u32(0) };
+
+ int i = 0;
+ do {
+ uint8x16_t s0 = vld1q_u8(src);
+ uint8x16_t s1 = vld1q_u8(src + src_stride);
+ uint8x16_t r0 = vld1q_u8(ref);
+ uint8x16_t r1 = vld1q_u8(ref + ref_stride);
+
+ uint8x16_t abs_diff0 = vabdq_u8(s0, r0);
+ uint8x16_t abs_diff1 = vabdq_u8(s1, r1);
+
+ sse_u32[0] = vdotq_u32(sse_u32[0], abs_diff0, abs_diff0);
+ sse_u32[1] = vdotq_u32(sse_u32[1], abs_diff1, abs_diff1);
+
+ src += 2 * src_stride;
+ ref += 2 * ref_stride;
+ i += 2;
+ } while (i < h);
+
+ *sse = horizontal_add_u32x4(vaddq_u32(sse_u32[0], sse_u32[1]));
+ return horizontal_add_u32x4(vaddq_u32(sse_u32[0], sse_u32[1]));
+}
+
+unsigned int aom_get4x4sse_cs_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride) {
+ uint8x16_t s = load_unaligned_u8q(src, src_stride);
+ uint8x16_t r = load_unaligned_u8q(ref, ref_stride);
+
+ uint8x16_t abs_diff = vabdq_u8(s, r);
+
+ uint32x4_t sse = vdotq_u32(vdupq_n_u32(0), abs_diff, abs_diff);
+
+ return horizontal_add_u32x4(sse);
+}
+
+#else // !defined(__ARM_FEATURE_DOTPROD)
+
+static INLINE unsigned int mse8xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ unsigned int *sse, int h) {
+ uint8x8_t s[2], r[2];
+ int16x4_t diff_lo[2], diff_hi[2];
+ uint16x8_t diff[2];
+ int32x4_t sse_s32[2] = { vdupq_n_s32(0), vdupq_n_s32(0) };
+
+ int i = 0;
+ do {
+ s[0] = vld1_u8(src);
+ src += src_stride;
+ s[1] = vld1_u8(src);
+ src += src_stride;
+ r[0] = vld1_u8(ref);
+ ref += ref_stride;
+ r[1] = vld1_u8(ref);
+ ref += ref_stride;
+
+ diff[0] = vsubl_u8(s[0], r[0]);
+ diff[1] = vsubl_u8(s[1], r[1]);
+
+ diff_lo[0] = vreinterpret_s16_u16(vget_low_u16(diff[0]));
+ diff_lo[1] = vreinterpret_s16_u16(vget_low_u16(diff[1]));
+ sse_s32[0] = vmlal_s16(sse_s32[0], diff_lo[0], diff_lo[0]);
+ sse_s32[1] = vmlal_s16(sse_s32[1], diff_lo[1], diff_lo[1]);
+
+ diff_hi[0] = vreinterpret_s16_u16(vget_high_u16(diff[0]));
+ diff_hi[1] = vreinterpret_s16_u16(vget_high_u16(diff[1]));
+ sse_s32[0] = vmlal_s16(sse_s32[0], diff_hi[0], diff_hi[0]);
+ sse_s32[1] = vmlal_s16(sse_s32[1], diff_hi[1], diff_hi[1]);
+
+ i += 2;
+ } while (i < h);
+
+ sse_s32[0] = vaddq_s32(sse_s32[0], sse_s32[1]);
+
+ *sse = horizontal_add_u32x4(vreinterpretq_u32_s32(sse_s32[0]));
+ return horizontal_add_u32x4(vreinterpretq_u32_s32(sse_s32[0]));
+}
+
+static INLINE unsigned int mse16xh_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride,
+ unsigned int *sse, int h) {
+ uint8x16_t s[2], r[2];
+ int16x4_t diff_lo[4], diff_hi[4];
+ uint16x8_t diff[4];
+ int32x4_t sse_s32[4] = { vdupq_n_s32(0), vdupq_n_s32(0), vdupq_n_s32(0),
+ vdupq_n_s32(0) };
+
+ int i = 0;
+ do {
+ s[0] = vld1q_u8(src);
+ src += src_stride;
+ s[1] = vld1q_u8(src);
+ src += src_stride;
+ r[0] = vld1q_u8(ref);
+ ref += ref_stride;
+ r[1] = vld1q_u8(ref);
+ ref += ref_stride;
+
+ diff[0] = vsubl_u8(vget_low_u8(s[0]), vget_low_u8(r[0]));
+ diff[1] = vsubl_u8(vget_high_u8(s[0]), vget_high_u8(r[0]));
+ diff[2] = vsubl_u8(vget_low_u8(s[1]), vget_low_u8(r[1]));
+ diff[3] = vsubl_u8(vget_high_u8(s[1]), vget_high_u8(r[1]));
+
+ diff_lo[0] = vreinterpret_s16_u16(vget_low_u16(diff[0]));
+ diff_lo[1] = vreinterpret_s16_u16(vget_low_u16(diff[1]));
+ sse_s32[0] = vmlal_s16(sse_s32[0], diff_lo[0], diff_lo[0]);
+ sse_s32[1] = vmlal_s16(sse_s32[1], diff_lo[1], diff_lo[1]);
+
+ diff_lo[2] = vreinterpret_s16_u16(vget_low_u16(diff[2]));
+ diff_lo[3] = vreinterpret_s16_u16(vget_low_u16(diff[3]));
+ sse_s32[2] = vmlal_s16(sse_s32[2], diff_lo[2], diff_lo[2]);
+ sse_s32[3] = vmlal_s16(sse_s32[3], diff_lo[3], diff_lo[3]);
+
+ diff_hi[0] = vreinterpret_s16_u16(vget_high_u16(diff[0]));
+ diff_hi[1] = vreinterpret_s16_u16(vget_high_u16(diff[1]));
+ sse_s32[0] = vmlal_s16(sse_s32[0], diff_hi[0], diff_hi[0]);
+ sse_s32[1] = vmlal_s16(sse_s32[1], diff_hi[1], diff_hi[1]);
+
+ diff_hi[2] = vreinterpret_s16_u16(vget_high_u16(diff[2]));
+ diff_hi[3] = vreinterpret_s16_u16(vget_high_u16(diff[3]));
+ sse_s32[2] = vmlal_s16(sse_s32[2], diff_hi[2], diff_hi[2]);
+ sse_s32[3] = vmlal_s16(sse_s32[3], diff_hi[3], diff_hi[3]);
+
+ i += 2;
+ } while (i < h);
+
+ sse_s32[0] = vaddq_s32(sse_s32[0], sse_s32[1]);
+ sse_s32[2] = vaddq_s32(sse_s32[2], sse_s32[3]);
+ sse_s32[0] = vaddq_s32(sse_s32[0], sse_s32[2]);
+
+ *sse = horizontal_add_u32x4(vreinterpretq_u32_s32(sse_s32[0]));
+ return horizontal_add_u32x4(vreinterpretq_u32_s32(sse_s32[0]));
+}
+
+unsigned int aom_get4x4sse_cs_neon(const uint8_t *src, int src_stride,
+ const uint8_t *ref, int ref_stride) {
+ uint8x8_t s[4], r[4];
+ int16x4_t diff[4];
+ int32x4_t sse;
+
+ s[0] = vld1_u8(src);
+ src += src_stride;
+ r[0] = vld1_u8(ref);
+ ref += ref_stride;
+ s[1] = vld1_u8(src);
+ src += src_stride;
+ r[1] = vld1_u8(ref);
+ ref += ref_stride;
+ s[2] = vld1_u8(src);
+ src += src_stride;
+ r[2] = vld1_u8(ref);
+ ref += ref_stride;
+ s[3] = vld1_u8(src);
+ r[3] = vld1_u8(ref);
+
+ diff[0] = vget_low_s16(vreinterpretq_s16_u16(vsubl_u8(s[0], r[0])));
+ diff[1] = vget_low_s16(vreinterpretq_s16_u16(vsubl_u8(s[1], r[1])));
+ diff[2] = vget_low_s16(vreinterpretq_s16_u16(vsubl_u8(s[2], r[2])));
+ diff[3] = vget_low_s16(vreinterpretq_s16_u16(vsubl_u8(s[3], r[3])));
+
+ sse = vmull_s16(diff[0], diff[0]);
+ sse = vmlal_s16(sse, diff[1], diff[1]);
+ sse = vmlal_s16(sse, diff[2], diff[2]);
+ sse = vmlal_s16(sse, diff[3], diff[3]);
+
+ return horizontal_add_u32x4(vreinterpretq_u32_s32(sse));
+}
+
+#endif // defined(__ARM_FEATURE_DOTPROD)
+
+#define MSE_WXH_NEON(w, h) \
+ unsigned int aom_mse##w##x##h##_neon(const uint8_t *src, int src_stride, \
+ const uint8_t *ref, int ref_stride, \
+ unsigned int *sse) { \
+ return mse##w##xh_neon(src, src_stride, ref, ref_stride, sse, h); \
+ }
+
+MSE_WXH_NEON(8, 8)
+MSE_WXH_NEON(8, 16)
+
+MSE_WXH_NEON(16, 8)
+MSE_WXH_NEON(16, 16)
+
+#undef MSE_WXH_NEON
diff --git a/aom_dsp/avg.c b/aom_dsp/avg.c
index 1e48bc1..a3821e6 100644
--- a/aom_dsp/avg.c
+++ b/aom_dsp/avg.c
@@ -547,6 +547,9 @@
}
// (mean * mean): dynamic range 31 bits.
- var = sse - ((mean * mean) >> (bwl + 2));
+ // If width == 128, the mean can be 510 * 128 = 65280, and log2(65280 ** 2) ~=
+ // 31.99, so it needs to be casted to unsigned int to compute its square.
+ const unsigned int mean_abs = mean >= 0 ? mean : -mean;
+ var = sse - ((mean_abs * mean_abs) >> (bwl + 2));
return var;
}
diff --git a/aom_dsp/intrapred.c b/aom_dsp/intrapred.c
index 00396c8..6ec091f 100644
--- a/aom_dsp/intrapred.c
+++ b/aom_dsp/intrapred.c
@@ -52,9 +52,9 @@
const int p_top_left = abs_diff(base, top_left);
// Return nearest to base of left, top and top_left.
- return (p_left <= p_top && p_left <= p_top_left)
- ? left
- : (p_top <= p_top_left) ? top : top_left;
+ return (p_left <= p_top && p_left <= p_top_left) ? left
+ : (p_top <= p_top_left) ? top
+ : top_left;
}
static INLINE void paeth_predictor(uint8_t *dst, ptrdiff_t stride, int bw,
diff --git a/aom_dsp/simd/v128_intrinsics_x86.h b/aom_dsp/simd/v128_intrinsics_x86.h
index c404015..32b51c9 100644
--- a/aom_dsp/simd/v128_intrinsics_x86.h
+++ b/aom_dsp/simd/v128_intrinsics_x86.h
@@ -36,7 +36,7 @@
}
SIMD_INLINE v128 v128_from_32(uint32_t a, uint32_t b, uint32_t c, uint32_t d) {
- return _mm_set_epi32(a, b, c, d);
+ return _mm_set_epi32((int)a, (int)b, (int)c, (int)d);
}
SIMD_INLINE v128 v128_load_aligned(const void *p) {
@@ -81,16 +81,16 @@
SIMD_INLINE v128 v128_zero() { return _mm_setzero_si128(); }
-SIMD_INLINE v128 v128_dup_8(uint8_t x) { return _mm_set1_epi8(x); }
+SIMD_INLINE v128 v128_dup_8(uint8_t x) { return _mm_set1_epi8((char)x); }
-SIMD_INLINE v128 v128_dup_16(uint16_t x) { return _mm_set1_epi16(x); }
+SIMD_INLINE v128 v128_dup_16(uint16_t x) { return _mm_set1_epi16((short)x); }
-SIMD_INLINE v128 v128_dup_32(uint32_t x) { return _mm_set1_epi32(x); }
+SIMD_INLINE v128 v128_dup_32(uint32_t x) { return _mm_set1_epi32((int)x); }
SIMD_INLINE v128 v128_dup_64(uint64_t x) {
// _mm_set_pi64x and _mm_cvtsi64x_si64 missing in some compilers
- return _mm_set_epi32((uint32_t)(x >> 32), (uint32_t)x, (uint32_t)(x >> 32),
- (uint32_t)x);
+ return _mm_set_epi32((int32_t)(x >> 32), (int32_t)x, (int32_t)(x >> 32),
+ (int32_t)x);
}
SIMD_INLINE v128 v128_add_8(v128 a, v128 b) { return _mm_add_epi8(a, b); }
@@ -304,7 +304,7 @@
v128 output;
unsigned char *input = (unsigned char *)&x;
unsigned char *index = (unsigned char *)&pattern;
- char *selected = (char *)&output;
+ unsigned char *selected = (unsigned char *)&output;
int counter;
for (counter = 0; counter < 16; counter++) {
@@ -534,58 +534,58 @@
SIMD_INLINE v128 v128_cmpeq_16(v128 a, v128 b) { return _mm_cmpeq_epi16(a, b); }
SIMD_INLINE v128 v128_shl_8(v128 a, unsigned int c) {
- return _mm_and_si128(_mm_set1_epi8((uint8_t)(0xff << c)),
- _mm_sll_epi16(a, _mm_cvtsi32_si128(c)));
+ return _mm_and_si128(_mm_set1_epi8((char)(0xff << c)),
+ _mm_sll_epi16(a, _mm_cvtsi32_si128((int)c)));
}
SIMD_INLINE v128 v128_shr_u8(v128 a, unsigned int c) {
return _mm_and_si128(_mm_set1_epi8((char)(0xff >> c)),
- _mm_srl_epi16(a, _mm_cvtsi32_si128(c)));
+ _mm_srl_epi16(a, _mm_cvtsi32_si128((int)c)));
}
SIMD_INLINE v128 v128_shr_s8(v128 a, unsigned int c) {
- __m128i x = _mm_cvtsi32_si128(c + 8);
+ __m128i x = _mm_cvtsi32_si128((int)(c + 8));
return _mm_packs_epi16(_mm_sra_epi16(_mm_unpacklo_epi8(a, a), x),
_mm_sra_epi16(_mm_unpackhi_epi8(a, a), x));
}
SIMD_INLINE v128 v128_shl_16(v128 a, unsigned int c) {
- return _mm_sll_epi16(a, _mm_cvtsi32_si128(c));
+ return _mm_sll_epi16(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v128 v128_shr_u16(v128 a, unsigned int c) {
- return _mm_srl_epi16(a, _mm_cvtsi32_si128(c));
+ return _mm_srl_epi16(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v128 v128_shr_s16(v128 a, unsigned int c) {
- return _mm_sra_epi16(a, _mm_cvtsi32_si128(c));
+ return _mm_sra_epi16(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v128 v128_shl_32(v128 a, unsigned int c) {
- return _mm_sll_epi32(a, _mm_cvtsi32_si128(c));
+ return _mm_sll_epi32(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v128 v128_shr_u32(v128 a, unsigned int c) {
- return _mm_srl_epi32(a, _mm_cvtsi32_si128(c));
+ return _mm_srl_epi32(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v128 v128_shr_s32(v128 a, unsigned int c) {
- return _mm_sra_epi32(a, _mm_cvtsi32_si128(c));
+ return _mm_sra_epi32(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v128 v128_shl_64(v128 a, unsigned int c) {
- return _mm_sll_epi64(a, _mm_cvtsi32_si128(c));
+ return _mm_sll_epi64(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v128 v128_shr_u64(v128 a, unsigned int c) {
- return _mm_srl_epi64(a, _mm_cvtsi32_si128(c));
+ return _mm_srl_epi64(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v128 v128_shr_s64(v128 a, unsigned int c) {
// _mm_sra_epi64 is missing in gcc?
- return v128_from_64((int64_t)v64_u64(v128_high_v64(a)) >> c,
- (int64_t)v64_u64(v128_low_v64(a)) >> c);
- // return _mm_sra_epi64(a, _mm_cvtsi32_si128(c));
+ return v128_from_64((uint64_t)((int64_t)v64_u64(v128_high_v64(a)) >> c),
+ (uint64_t)((int64_t)v64_u64(v128_low_v64(a)) >> c));
+ // return _mm_sra_epi64(a, _mm_cvtsi32_si128((int)c));
}
/* These intrinsics require immediate values, so we must use #defines
@@ -593,9 +593,9 @@
#define v128_shl_n_byte(a, c) _mm_slli_si128(a, (c)&127)
#define v128_shr_n_byte(a, c) _mm_srli_si128(a, (c)&127)
#define v128_shl_n_8(a, c) \
- _mm_and_si128(_mm_set1_epi8((uint8_t)(0xff << (c))), _mm_slli_epi16(a, c))
+ _mm_and_si128(_mm_set1_epi8((char)(0xff << (c))), _mm_slli_epi16(a, c))
#define v128_shr_n_u8(a, c) \
- _mm_and_si128(_mm_set1_epi8(0xff >> (c)), _mm_srli_epi16(a, c))
+ _mm_and_si128(_mm_set1_epi8((char)(0xff >> (c))), _mm_srli_epi16(a, c))
#define v128_shr_n_s8(a, c) \
_mm_packs_epi16(_mm_srai_epi16(_mm_unpacklo_epi8(a, a), (c) + 8), \
_mm_srai_epi16(_mm_unpackhi_epi8(a, a), (c) + 8))
diff --git a/aom_dsp/simd/v256_intrinsics_c.h b/aom_dsp/simd/v256_intrinsics_c.h
index 47003af..66cfda3 100644
--- a/aom_dsp/simd/v256_intrinsics_c.h
+++ b/aom_dsp/simd/v256_intrinsics_c.h
@@ -380,7 +380,7 @@
}
SIMD_INLINE uint32_t c_v256_movemask_8(c_v256 a) {
- return ((a.s8[31] < 0) << 31) | ((a.s8[30] < 0) << 30) |
+ return ((uint32_t)(a.s8[31] < 0) << 31) | ((a.s8[30] < 0) << 30) |
((a.s8[29] < 0) << 29) | ((a.s8[28] < 0) << 28) |
((a.s8[27] < 0) << 27) | ((a.s8[26] < 0) << 26) |
((a.s8[25] < 0) << 25) | ((a.s8[24] < 0) << 24) |
diff --git a/aom_dsp/simd/v256_intrinsics_x86.h b/aom_dsp/simd/v256_intrinsics_x86.h
index eb5eaf0..894ddee 100644
--- a/aom_dsp/simd/v256_intrinsics_x86.h
+++ b/aom_dsp/simd/v256_intrinsics_x86.h
@@ -57,7 +57,7 @@
}
SIMD_INLINE v256 v256_from_64(uint64_t a, uint64_t b, uint64_t c, uint64_t d) {
- return _mm256_set_epi64x(a, b, c, d);
+ return _mm256_set_epi64x((int64_t)a, (int64_t)b, (int64_t)c, (int64_t)d);
}
SIMD_INLINE v256 v256_load_aligned(const void *p) {
@@ -78,13 +78,15 @@
SIMD_INLINE v256 v256_zero(void) { return _mm256_setzero_si256(); }
-SIMD_INLINE v256 v256_dup_8(uint8_t x) { return _mm256_set1_epi8(x); }
+SIMD_INLINE v256 v256_dup_8(uint8_t x) { return _mm256_set1_epi8((char)x); }
-SIMD_INLINE v256 v256_dup_16(uint16_t x) { return _mm256_set1_epi16(x); }
+SIMD_INLINE v256 v256_dup_16(uint16_t x) { return _mm256_set1_epi16((short)x); }
-SIMD_INLINE v256 v256_dup_32(uint32_t x) { return _mm256_set1_epi32(x); }
+SIMD_INLINE v256 v256_dup_32(uint32_t x) { return _mm256_set1_epi32((int)x); }
-SIMD_INLINE v256 v256_dup_64(uint64_t x) { return _mm256_set1_epi64x(x); }
+SIMD_INLINE v256 v256_dup_64(uint64_t x) {
+ return _mm256_set1_epi64x((int64_t)x);
+}
SIMD_INLINE v256 v256_add_8(v256 a, v256 b) { return _mm256_add_epi8(a, b); }
@@ -543,7 +545,9 @@
SIMD_INLINE v256 v256_min_s8(v256 a, v256 b) { return _mm256_min_epi8(a, b); }
-SIMD_INLINE uint32_t v256_movemask_8(v256 a) { return _mm256_movemask_epi8(a); }
+SIMD_INLINE uint32_t v256_movemask_8(v256 a) {
+ return (uint32_t)_mm256_movemask_epi8(a);
+}
SIMD_INLINE v256 v256_blend_8(v256 a, v256 b, v256 c) {
return _mm256_blendv_epi8(a, b, c);
@@ -596,56 +600,56 @@
}
SIMD_INLINE v256 v256_shl_8(v256 a, unsigned int c) {
- return _mm256_and_si256(_mm256_set1_epi8((uint8_t)(0xff << c)),
- _mm256_sll_epi16(a, _mm_cvtsi32_si128(c)));
+ return _mm256_and_si256(_mm256_set1_epi8((char)(0xff << c)),
+ _mm256_sll_epi16(a, _mm_cvtsi32_si128((int)c)));
}
SIMD_INLINE v256 v256_shr_u8(v256 a, unsigned int c) {
return _mm256_and_si256(_mm256_set1_epi8((char)(0xff >> c)),
- _mm256_srl_epi16(a, _mm_cvtsi32_si128(c)));
+ _mm256_srl_epi16(a, _mm_cvtsi32_si128((int)c)));
}
SIMD_INLINE v256 v256_shr_s8(v256 a, unsigned int c) {
- __m128i x = _mm_cvtsi32_si128(c + 8);
+ __m128i x = _mm_cvtsi32_si128((int)(c + 8));
return _mm256_packs_epi16(_mm256_sra_epi16(_mm256_unpacklo_epi8(a, a), x),
_mm256_sra_epi16(_mm256_unpackhi_epi8(a, a), x));
}
SIMD_INLINE v256 v256_shl_16(v256 a, unsigned int c) {
- return _mm256_sll_epi16(a, _mm_cvtsi32_si128(c));
+ return _mm256_sll_epi16(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v256 v256_shr_u16(v256 a, unsigned int c) {
- return _mm256_srl_epi16(a, _mm_cvtsi32_si128(c));
+ return _mm256_srl_epi16(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v256 v256_shr_s16(v256 a, unsigned int c) {
- return _mm256_sra_epi16(a, _mm_cvtsi32_si128(c));
+ return _mm256_sra_epi16(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v256 v256_shl_32(v256 a, unsigned int c) {
- return _mm256_sll_epi32(a, _mm_cvtsi32_si128(c));
+ return _mm256_sll_epi32(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v256 v256_shr_u32(v256 a, unsigned int c) {
- return _mm256_srl_epi32(a, _mm_cvtsi32_si128(c));
+ return _mm256_srl_epi32(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v256 v256_shr_s32(v256 a, unsigned int c) {
- return _mm256_sra_epi32(a, _mm_cvtsi32_si128(c));
+ return _mm256_sra_epi32(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v256 v256_shl_64(v256 a, unsigned int c) {
- return _mm256_sll_epi64(a, _mm_cvtsi32_si128(c));
+ return _mm256_sll_epi64(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v256 v256_shr_u64(v256 a, unsigned int c) {
- return _mm256_srl_epi64(a, _mm_cvtsi32_si128(c));
+ return _mm256_srl_epi64(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v256 v256_shr_s64(v256 a, unsigned int c) {
#if defined(__AVX512VL__)
- return _mm256_sra_epi64(a, _mm_cvtsi32_si128(c));
+ return _mm256_sra_epi64(a, _mm_cvtsi32_si128((int)c));
#else
return v256_from_v128(v128_shr_s64(v256_high_v128(a), c),
v128_shr_s64(v256_low_v128(a), c));
@@ -677,11 +681,12 @@
#define v256_align(a, b, c) \
((c) ? v256_or(v256_shr_n_byte(b, c), v256_shl_n_byte(a, 32 - (c))) : b)
-#define v256_shl_n_8(a, c) \
- _mm256_and_si256(_mm256_set1_epi8((uint8_t)(0xff << (c))), \
+#define v256_shl_n_8(a, c) \
+ _mm256_and_si256(_mm256_set1_epi8((char)(0xff << (c))), \
_mm256_slli_epi16(a, c))
-#define v256_shr_n_u8(a, c) \
- _mm256_and_si256(_mm256_set1_epi8(0xff >> (c)), _mm256_srli_epi16(a, c))
+#define v256_shr_n_u8(a, c) \
+ _mm256_and_si256(_mm256_set1_epi8((char)(0xff >> (c))), \
+ _mm256_srli_epi16(a, c))
#define v256_shr_n_s8(a, c) \
_mm256_packs_epi16(_mm256_srai_epi16(_mm256_unpacklo_epi8(a, a), (c) + 8), \
_mm256_srai_epi16(_mm256_unpackhi_epi8(a, a), (c) + 8))
diff --git a/aom_dsp/simd/v64_intrinsics_c.h b/aom_dsp/simd/v64_intrinsics_c.h
index b84f243..bfd6fe0 100644
--- a/aom_dsp/simd/v64_intrinsics_c.h
+++ b/aom_dsp/simd/v64_intrinsics_c.h
@@ -186,11 +186,7 @@
c_v64 t;
int c;
for (c = 0; c < 8; c++)
- t.u8[c] = (int16_t)a.u8[c] + (int16_t)b.u8[c] > 255
- ? 255
- : (int16_t)a.u8[c] + (int16_t)b.u8[c] < 0
- ? 0
- : (int16_t)a.u8[c] + (int16_t)b.u8[c];
+ t.u8[c] = SIMD_CLAMP((int16_t)a.u8[c] + (int16_t)b.u8[c], 0, 255);
return t;
}
@@ -198,11 +194,7 @@
c_v64 t;
int c;
for (c = 0; c < 8; c++)
- t.s8[c] = (int16_t)a.s8[c] + (int16_t)b.s8[c] > 127
- ? 127
- : (int16_t)a.s8[c] + (int16_t)b.s8[c] < -128
- ? -128
- : (int16_t)a.s8[c] + (int16_t)b.s8[c];
+ t.s8[c] = SIMD_CLAMP((int16_t)a.s8[c] + (int16_t)b.s8[c], -128, 127);
return t;
}
@@ -210,11 +202,7 @@
c_v64 t;
int c;
for (c = 0; c < 4; c++)
- t.s16[c] = (int32_t)a.s16[c] + (int32_t)b.s16[c] > 32767
- ? 32767
- : (int32_t)a.s16[c] + (int32_t)b.s16[c] < -32768
- ? -32768
- : (int32_t)a.s16[c] + (int32_t)b.s16[c];
+ t.s16[c] = SIMD_CLAMP((int32_t)a.s16[c] + (int32_t)b.s16[c], -32768, 32767);
return t;
}
@@ -244,7 +232,7 @@
int c;
for (c = 0; c < 8; c++) {
int16_t d = (int16_t)a.s8[c] - (int16_t)b.s8[c];
- t.s8[c] = d > 127 ? 127 : (d < -128 ? -128 : d);
+ t.s8[c] = SIMD_CLAMP(d, -128, 127);
}
return t;
}
@@ -260,11 +248,7 @@
c_v64 t;
int c;
for (c = 0; c < 4; c++)
- t.s16[c] = (int32_t)a.s16[c] - (int32_t)b.s16[c] < -32768
- ? -32768
- : (int32_t)a.s16[c] - (int32_t)b.s16[c] > 32767
- ? 32767
- : (int32_t)a.s16[c] - (int32_t)b.s16[c];
+ t.s16[c] = SIMD_CLAMP((int32_t)a.s16[c] - (int32_t)b.s16[c], -32768, 32767);
return t;
}
@@ -481,10 +465,10 @@
a = b;
b = u;
}
- t.s16[3] = a.s32[1] > 32767 ? 32767 : a.s32[1] < -32768 ? -32768 : a.s32[1];
- t.s16[2] = a.s32[0] > 32767 ? 32767 : a.s32[0] < -32768 ? -32768 : a.s32[0];
- t.s16[1] = b.s32[1] > 32767 ? 32767 : b.s32[1] < -32768 ? -32768 : b.s32[1];
- t.s16[0] = b.s32[0] > 32767 ? 32767 : b.s32[0] < -32768 ? -32768 : b.s32[0];
+ t.s16[3] = SIMD_CLAMP(a.s32[1], -32768, 32767);
+ t.s16[2] = SIMD_CLAMP(a.s32[0], -32768, 32767);
+ t.s16[1] = SIMD_CLAMP(b.s32[1], -32768, 32767);
+ t.s16[0] = SIMD_CLAMP(b.s32[0], -32768, 32767);
return t;
}
@@ -495,10 +479,10 @@
a = b;
b = u;
}
- t.u16[3] = a.s32[1] > 65535 ? 65535 : a.s32[1] < 0 ? 0 : a.s32[1];
- t.u16[2] = a.s32[0] > 65535 ? 65535 : a.s32[0] < 0 ? 0 : a.s32[0];
- t.u16[1] = b.s32[1] > 65535 ? 65535 : b.s32[1] < 0 ? 0 : b.s32[1];
- t.u16[0] = b.s32[0] > 65535 ? 65535 : b.s32[0] < 0 ? 0 : b.s32[0];
+ t.u16[3] = SIMD_CLAMP(a.s32[1], 0, 65535);
+ t.u16[2] = SIMD_CLAMP(a.s32[0], 0, 65535);
+ t.u16[1] = SIMD_CLAMP(b.s32[1], 0, 65535);
+ t.u16[0] = SIMD_CLAMP(b.s32[0], 0, 65535);
return t;
}
@@ -509,14 +493,14 @@
a = b;
b = u;
}
- t.u8[7] = a.s16[3] > 255 ? 255 : a.s16[3] < 0 ? 0 : a.s16[3];
- t.u8[6] = a.s16[2] > 255 ? 255 : a.s16[2] < 0 ? 0 : a.s16[2];
- t.u8[5] = a.s16[1] > 255 ? 255 : a.s16[1] < 0 ? 0 : a.s16[1];
- t.u8[4] = a.s16[0] > 255 ? 255 : a.s16[0] < 0 ? 0 : a.s16[0];
- t.u8[3] = b.s16[3] > 255 ? 255 : b.s16[3] < 0 ? 0 : b.s16[3];
- t.u8[2] = b.s16[2] > 255 ? 255 : b.s16[2] < 0 ? 0 : b.s16[2];
- t.u8[1] = b.s16[1] > 255 ? 255 : b.s16[1] < 0 ? 0 : b.s16[1];
- t.u8[0] = b.s16[0] > 255 ? 255 : b.s16[0] < 0 ? 0 : b.s16[0];
+ t.u8[7] = SIMD_CLAMP(a.s16[3], 0, 255);
+ t.u8[6] = SIMD_CLAMP(a.s16[2], 0, 255);
+ t.u8[5] = SIMD_CLAMP(a.s16[1], 0, 255);
+ t.u8[4] = SIMD_CLAMP(a.s16[0], 0, 255);
+ t.u8[3] = SIMD_CLAMP(b.s16[3], 0, 255);
+ t.u8[2] = SIMD_CLAMP(b.s16[2], 0, 255);
+ t.u8[1] = SIMD_CLAMP(b.s16[1], 0, 255);
+ t.u8[0] = SIMD_CLAMP(b.s16[0], 0, 255);
return t;
}
@@ -527,14 +511,14 @@
a = b;
b = u;
}
- t.u8[7] = (uint8_t)(a.s16[3] > 127 ? 127 : a.s16[3] < -128 ? 128 : a.s16[3]);
- t.u8[6] = (uint8_t)(a.s16[2] > 127 ? 127 : a.s16[2] < -128 ? 128 : a.s16[2]);
- t.u8[5] = (uint8_t)(a.s16[1] > 127 ? 127 : a.s16[1] < -128 ? 128 : a.s16[1]);
- t.u8[4] = (uint8_t)(a.s16[0] > 127 ? 127 : a.s16[0] < -128 ? 128 : a.s16[0]);
- t.u8[3] = (uint8_t)(b.s16[3] > 127 ? 127 : b.s16[3] < -128 ? 128 : b.s16[3]);
- t.u8[2] = (uint8_t)(b.s16[2] > 127 ? 127 : b.s16[2] < -128 ? 128 : b.s16[2]);
- t.u8[1] = (uint8_t)(b.s16[1] > 127 ? 127 : b.s16[1] < -128 ? 128 : b.s16[1]);
- t.u8[0] = (uint8_t)(b.s16[0] > 127 ? 127 : b.s16[0] < -128 ? 128 : b.s16[0]);
+ t.s8[7] = SIMD_CLAMP(a.s16[3], -128, 127);
+ t.s8[6] = SIMD_CLAMP(a.s16[2], -128, 127);
+ t.s8[5] = SIMD_CLAMP(a.s16[1], -128, 127);
+ t.s8[4] = SIMD_CLAMP(a.s16[0], -128, 127);
+ t.s8[3] = SIMD_CLAMP(b.s16[3], -128, 127);
+ t.s8[2] = SIMD_CLAMP(b.s16[2], -128, 127);
+ t.s8[1] = SIMD_CLAMP(b.s16[1], -128, 127);
+ t.s8[0] = SIMD_CLAMP(b.s16[0], -128, 127);
return t;
}
@@ -702,13 +686,13 @@
c_v64 t;
int32_t u;
u = a.u8[0] * b.s8[0] + a.u8[1] * b.s8[1];
- t.s16[0] = u > 32767 ? 32767 : u < -32768 ? -32768 : u;
+ t.s16[0] = SIMD_CLAMP(u, -32768, 32767);
u = a.u8[2] * b.s8[2] + a.u8[3] * b.s8[3];
- t.s16[1] = u > 32767 ? 32767 : u < -32768 ? -32768 : u;
+ t.s16[1] = SIMD_CLAMP(u, -32768, 32767);
u = a.u8[4] * b.s8[4] + a.u8[5] * b.s8[5];
- t.s16[2] = u > 32767 ? 32767 : u < -32768 ? -32768 : u;
+ t.s16[2] = SIMD_CLAMP(u, -32768, 32767);
u = a.u8[6] * b.s8[6] + a.u8[7] * b.s8[7];
- t.s16[3] = u > 32767 ? 32767 : u < -32768 ? -32768 : u;
+ t.s16[3] = SIMD_CLAMP(u, -32768, 32767);
return t;
}
diff --git a/aom_dsp/simd/v64_intrinsics_x86.h b/aom_dsp/simd/v64_intrinsics_x86.h
index 1f273fe..ec27a6b 100644
--- a/aom_dsp/simd/v64_intrinsics_x86.h
+++ b/aom_dsp/simd/v64_intrinsics_x86.h
@@ -43,14 +43,14 @@
}
SIMD_INLINE v64 v64_from_32(uint32_t x, uint32_t y) {
- return _mm_set_epi32(0, 0, x, y);
+ return _mm_set_epi32(0, 0, (int32_t)x, (int32_t)y);
}
SIMD_INLINE v64 v64_from_64(uint64_t x) {
#ifdef __x86_64__
- return _mm_cvtsi64_si128(x);
+ return _mm_cvtsi64_si128((int64_t)x);
#else
- return _mm_set_epi32(0, 0, x >> 32, (uint32_t)x);
+ return _mm_set_epi32(0, 0, (int32_t)(x >> 32), (int32_t)x);
#endif
}
@@ -101,11 +101,11 @@
SIMD_INLINE v64 v64_zero(void) { return _mm_setzero_si128(); }
-SIMD_INLINE v64 v64_dup_8(uint8_t x) { return _mm_set1_epi8(x); }
+SIMD_INLINE v64 v64_dup_8(uint8_t x) { return _mm_set1_epi8((char)x); }
-SIMD_INLINE v64 v64_dup_16(uint16_t x) { return _mm_set1_epi16(x); }
+SIMD_INLINE v64 v64_dup_16(uint16_t x) { return _mm_set1_epi16((short)x); }
-SIMD_INLINE v64 v64_dup_32(uint32_t x) { return _mm_set1_epi32(x); }
+SIMD_INLINE v64 v64_dup_32(uint32_t x) { return _mm_set1_epi32((int)x); }
SIMD_INLINE v64 v64_add_8(v64 a, v64 b) { return _mm_add_epi8(a, b); }
@@ -178,14 +178,11 @@
__m128i t = _mm_unpacklo_epi64(b, a);
return _mm_packus_epi32(t, t);
#else
- int32_t ah = v64_high_u32(a);
- int32_t al = v64_low_u32(a);
- int32_t bh = v64_high_u32(b);
- int32_t bl = v64_low_u32(b);
- return v64_from_16(ah > 65535 ? 65535 : ah < 0 ? 0 : ah,
- al > 65535 ? 65535 : al < 0 ? 0 : al,
- bh > 65535 ? 65535 : bh < 0 ? 0 : bh,
- bl > 65535 ? 65535 : bl < 0 ? 0 : bl);
+ const int32_t ah = SIMD_CLAMP(v64_high_s32(a), 0, 65535);
+ const int32_t al = SIMD_CLAMP(v64_low_s32(a), 0, 65535);
+ const int32_t bh = SIMD_CLAMP(v64_high_s32(b), 0, 65535);
+ const int32_t bl = SIMD_CLAMP(v64_low_s32(b), 0, 65535);
+ return v64_from_16(ah, al, bh, bl);
#endif
}
@@ -279,7 +276,7 @@
v64 output;
unsigned char *input = (unsigned char *)&x;
unsigned char *index = (unsigned char *)&pattern;
- char *selected = (char *)&output;
+ unsigned char *selected = (unsigned char *)&output;
int counter;
for (counter = 0; counter < 8; counter++) {
@@ -433,42 +430,43 @@
SIMD_INLINE v64 v64_cmpeq_16(v64 a, v64 b) { return _mm_cmpeq_epi16(a, b); }
SIMD_INLINE v64 v64_shl_8(v64 a, unsigned int c) {
- return _mm_and_si128(_mm_set1_epi8((uint8_t)(0xff << c)),
- _mm_sll_epi16(a, _mm_cvtsi32_si128(c)));
+ return _mm_and_si128(_mm_set1_epi8((char)(0xff << c)),
+ _mm_sll_epi16(a, _mm_cvtsi32_si128((int)c)));
}
SIMD_INLINE v64 v64_shr_u8(v64 a, unsigned int c) {
return _mm_and_si128(_mm_set1_epi8((char)(0xff >> c)),
- _mm_srl_epi16(a, _mm_cvtsi32_si128(c)));
+ _mm_srl_epi16(a, _mm_cvtsi32_si128((int)c)));
}
SIMD_INLINE v64 v64_shr_s8(v64 a, unsigned int c) {
return _mm_packs_epi16(
- _mm_sra_epi16(_mm_unpacklo_epi8(a, a), _mm_cvtsi32_si128(c + 8)), a);
+ _mm_sra_epi16(_mm_unpacklo_epi8(a, a), _mm_cvtsi32_si128((int)(c + 8))),
+ a);
}
SIMD_INLINE v64 v64_shl_16(v64 a, unsigned int c) {
- return _mm_sll_epi16(a, _mm_cvtsi32_si128(c));
+ return _mm_sll_epi16(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v64 v64_shr_u16(v64 a, unsigned int c) {
- return _mm_srl_epi16(a, _mm_cvtsi32_si128(c));
+ return _mm_srl_epi16(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v64 v64_shr_s16(v64 a, unsigned int c) {
- return _mm_sra_epi16(a, _mm_cvtsi32_si128(c));
+ return _mm_sra_epi16(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v64 v64_shl_32(v64 a, unsigned int c) {
- return _mm_sll_epi32(a, _mm_cvtsi32_si128(c));
+ return _mm_sll_epi32(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v64 v64_shr_u32(v64 a, unsigned int c) {
- return _mm_srl_epi32(a, _mm_cvtsi32_si128(c));
+ return _mm_srl_epi32(a, _mm_cvtsi32_si128((int)c));
}
SIMD_INLINE v64 v64_shr_s32(v64 a, unsigned int c) {
- return _mm_sra_epi32(a, _mm_cvtsi32_si128(c));
+ return _mm_sra_epi32(a, _mm_cvtsi32_si128((int)c));
}
/* These intrinsics require immediate values, so we must use #defines
@@ -476,9 +474,9 @@
#define v64_shl_n_byte(a, c) _mm_slli_si128(a, c)
#define v64_shr_n_byte(a, c) _mm_srli_si128(_mm_unpacklo_epi64(a, a), c + 8)
#define v64_shl_n_8(a, c) \
- _mm_and_si128(_mm_set1_epi8((uint8_t)(0xff << (c))), _mm_slli_epi16(a, c))
+ _mm_and_si128(_mm_set1_epi8((char)(0xff << (c))), _mm_slli_epi16(a, c))
#define v64_shr_n_u8(a, c) \
- _mm_and_si128(_mm_set1_epi8(0xff >> (c)), _mm_srli_epi16(a, c))
+ _mm_and_si128(_mm_set1_epi8((char)(0xff >> (c))), _mm_srli_epi16(a, c))
#define v64_shr_n_s8(a, c) \
_mm_packs_epi16(_mm_srai_epi16(_mm_unpacklo_epi8(a, a), (c) + 8), a)
#define v64_shl_n_16(a, c) _mm_slli_epi16(a, c)
diff --git a/aom_dsp/x86/avg_intrin_sse4.c b/aom_dsp/x86/avg_intrin_sse4.c
new file mode 100644
index 0000000..55a483e
--- /dev/null
+++ b/aom_dsp/x86/avg_intrin_sse4.c
@@ -0,0 +1,60 @@
+/*
+ * Copyright (c) 2022, Alliance for Open Media. All rights reserved
+ *
+ * This source code is subject to the terms of the BSD 2 Clause License and
+ * the Alliance for Open Media Patent License 1.0. If the BSD 2 Clause License
+ * was not distributed with this source code in the LICENSE file, you can
+ * obtain it at www.aomedia.org/license/software. If the Alliance for Open
+ * Media Patent License 1.0 was not distributed with this source code in the
+ * PATENTS file, you can obtain it at www.aomedia.org/license/patent.
+ */
+
+#include <smmintrin.h>
+
+#include "config/aom_dsp_rtcd.h"
+
+// ref: [0 - 510]
+// src: [0 - 510]
+// bwl: {2, 3, 4, 5}
+int aom_vector_var_sse4_1(const int16_t *ref, const int16_t *src,
+ const int log_bw) {
+ const int width = 4 << log_bw;
+ assert(width % 16 == 0);
+
+ const __m128i k_one_epi16 = _mm_set1_epi16((int16_t)1);
+ __m128i mean = _mm_setzero_si128();
+ __m128i sse = _mm_setzero_si128();
+
+ for (int i = 0; i < width; i += 16) {
+ const __m128i src_line = _mm_loadu_si128((const __m128i *)src);
+ const __m128i ref_line = _mm_loadu_si128((const __m128i *)ref);
+ const __m128i src_line2 = _mm_loadu_si128((const __m128i *)(src + 8));
+ const __m128i ref_line2 = _mm_loadu_si128((const __m128i *)(ref + 8));
+ __m128i diff = _mm_sub_epi16(ref_line, src_line);
+ const __m128i diff2 = _mm_sub_epi16(ref_line2, src_line2);
+ __m128i diff_sqr = _mm_madd_epi16(diff, diff);
+ const __m128i diff_sqr2 = _mm_madd_epi16(diff2, diff2);
+
+ diff = _mm_add_epi16(diff, diff2);
+ diff_sqr = _mm_add_epi32(diff_sqr, diff_sqr2);
+ diff = _mm_madd_epi16(diff, k_one_epi16);
+ sse = _mm_add_epi32(sse, diff_sqr);
+
+ mean = _mm_add_epi32(mean, diff);
+
+ src += 16;
+ ref += 16;
+ }
+
+ mean = _mm_hadd_epi32(mean, mean);
+ sse = _mm_hadd_epi32(sse, sse);
+ mean = _mm_hadd_epi32(mean, mean);
+ sse = _mm_hadd_epi32(sse, sse);
+
+ // (mean * mean): dynamic range 31 bits.
+ const int mean_int = _mm_extract_epi32(mean, 0);
+ const int sse_int = _mm_extract_epi32(sse, 0);
+ const unsigned int mean_abs = mean_int >= 0 ? mean_int : -mean_int;
+ const int var = sse_int - ((mean_abs * mean_abs) >> (log_bw + 2));
+ return var;
+}
diff --git a/aom_dsp/x86/blend_a64_mask_avx2.c b/aom_dsp/x86/blend_a64_mask_avx2.c
index 95383d2..dfbab32 100644
--- a/aom_dsp/x86/blend_a64_mask_avx2.c
+++ b/aom_dsp/x86/blend_a64_mask_avx2.c
@@ -910,14 +910,14 @@
const __m256i *round_offset, int shift, const __m256i *clip_low,
const __m256i *clip_high, const __m256i *mask_max) {
// Load 4x u16 pixels from each of 4 rows from each source
- const __m256i s0 = _mm256_set_epi64x(*(uint64_t *)(src0 + 3 * src0_stride),
- *(uint64_t *)(src0 + 2 * src0_stride),
- *(uint64_t *)(src0 + 1 * src0_stride),
- *(uint64_t *)(src0 + 0 * src0_stride));
- const __m256i s1 = _mm256_set_epi64x(*(uint64_t *)(src1 + 3 * src1_stride),
- *(uint64_t *)(src1 + 2 * src1_stride),
- *(uint64_t *)(src1 + 1 * src1_stride),
- *(uint64_t *)(src1 + 0 * src1_stride));
+ const __m256i s0 = _mm256_set_epi64x(*(int64_t *)(src0 + 3 * src0_stride),
+ *(int64_t *)(src0 + 2 * src0_stride),
+ *(int64_t *)(src0 + 1 * src0_stride),
+ *(int64_t *)(src0 + 0 * src0_stride));
+ const __m256i s1 = _mm256_set_epi64x(*(int64_t *)(src1 + 3 * src1_stride),
+ *(int64_t *)(src1 + 2 * src1_stride),
+ *(int64_t *)(src1 + 1 * src1_stride),
+ *(int64_t *)(src1 + 0 * src1_stride));
// Generate the inverse mask
const __m256i mask1 = _mm256_sub_epi16(*mask_max, *mask0);
@@ -964,10 +964,10 @@
const __m256i *clip_high, const __m256i *mask_max) {
do {
// Load 8x u8 pixels from each of 4 rows of the mask, pad each to u16
- const __m128i mask08 = _mm_set_epi32(*(uint32_t *)(mask + 3 * mask_stride),
- *(uint32_t *)(mask + 2 * mask_stride),
- *(uint32_t *)(mask + 1 * mask_stride),
- *(uint32_t *)(mask + 0 * mask_stride));
+ const __m128i mask08 = _mm_set_epi32(*(int32_t *)(mask + 3 * mask_stride),
+ *(int32_t *)(mask + 2 * mask_stride),
+ *(int32_t *)(mask + 1 * mask_stride),
+ *(int32_t *)(mask + 0 * mask_stride));
const __m256i mask0 = _mm256_cvtepu8_epi16(mask08);
highbd_blend_a64_d16_mask_w4_avx2(dst, dst_stride, src0, src0_stride, src1,
@@ -994,15 +994,15 @@
// (saturating) add together rows then use madd to add adjacent pixels
// Finally, divide each value by 4 (with rounding)
const __m256i m0246 =
- _mm256_set_epi64x(*(uint64_t *)(mask + 6 * mask_stride),
- *(uint64_t *)(mask + 4 * mask_stride),
- *(uint64_t *)(mask + 2 * mask_stride),
- *(uint64_t *)(mask + 0 * mask_stride));
+ _mm256_set_epi64x(*(int64_t *)(mask + 6 * mask_stride),
+ *(int64_t *)(mask + 4 * mask_stride),
+ *(int64_t *)(mask + 2 * mask_stride),
+ *(int64_t *)(mask + 0 * mask_stride));
const __m256i m1357 =
- _mm256_set_epi64x(*(uint64_t *)(mask + 7 * mask_stride),
- *(uint64_t *)(mask + 5 * mask_stride),
- *(uint64_t *)(mask + 3 * mask_stride),
- *(uint64_t *)(mask + 1 * mask_stride));
+ _mm256_set_epi64x(*(int64_t *)(mask + 7 * mask_stride),
+ *(int64_t *)(mask + 5 * mask_stride),
+ *(int64_t *)(mask + 3 * mask_stride),
+ *(int64_t *)(mask + 1 * mask_stride));
const __m256i addrows = _mm256_adds_epu8(m0246, m1357);
const __m256i adjacent = _mm256_maddubs_epi16(addrows, one_b);
const __m256i mask0 =
@@ -1101,10 +1101,10 @@
do {
// Load 8x u8 pixels from each of 4 rows in the mask
const __m128i mask0a8 =
- _mm_set_epi64x(*(uint64_t *)mask, *(uint64_t *)(mask + mask_stride));
+ _mm_set_epi64x(*(int64_t *)mask, *(uint64_t *)(mask + mask_stride));
const __m128i mask0b8 =
- _mm_set_epi64x(*(uint64_t *)(mask + 2 * mask_stride),
- *(uint64_t *)(mask + 3 * mask_stride));
+ _mm_set_epi64x(*(int64_t *)(mask + 2 * mask_stride),
+ *(int64_t *)(mask + 3 * mask_stride));
const __m256i mask0a = _mm256_cvtepu8_epi16(mask0a8);
const __m256i mask0b = _mm256_cvtepu8_epi16(mask0b8);
@@ -1307,7 +1307,7 @@
const __m256i v_round_offset = _mm256_set1_epi32(round_offset);
const int shift = round_bits + AOM_BLEND_A64_ROUND_BITS;
- const __m256i clip_low = _mm256_set1_epi16(0);
+ const __m256i clip_low = _mm256_setzero_si256();
const __m256i clip_high = _mm256_set1_epi16((1 << bd) - 1);
const __m256i mask_max = _mm256_set1_epi16(AOM_BLEND_A64_MAX_ALPHA);
diff --git a/aom_dsp/x86/blend_a64_mask_sse4.c b/aom_dsp/x86/blend_a64_mask_sse4.c
index 4a368ef..58a7345 100644
--- a/aom_dsp/x86/blend_a64_mask_sse4.c
+++ b/aom_dsp/x86/blend_a64_mask_sse4.c
@@ -1121,13 +1121,13 @@
const __m128i *mask_max) {
// Load 4 pixels from each of 4 rows from each source
const __m128i s0a =
- _mm_set_epi64x(*(uint64_t *)src0, *(uint64_t *)(src0 + src0_stride));
- const __m128i s0b = _mm_set_epi64x(*(uint64_t *)(src0 + 2 * src0_stride),
- *(uint64_t *)(src0 + 3 * src0_stride));
+ _mm_set_epi64x(*(int64_t *)src0, *(int64_t *)(src0 + src0_stride));
+ const __m128i s0b = _mm_set_epi64x(*(int64_t *)(src0 + 2 * src0_stride),
+ *(int64_t *)(src0 + 3 * src0_stride));
const __m128i s1a =
- _mm_set_epi64x(*(uint64_t *)(src1), *(uint64_t *)(src1 + src1_stride));
- const __m128i s1b = _mm_set_epi64x(*(uint64_t *)(src1 + 2 * src1_stride),
- *(uint64_t *)(src1 + 3 * src1_stride));
+ _mm_set_epi64x(*(int64_t *)(src1), *(int64_t *)(src1 + src1_stride));
+ const __m128i s1b = _mm_set_epi64x(*(int64_t *)(src1 + 2 * src1_stride),
+ *(int64_t *)(src1 + 3 * src1_stride));
// Generate the inverse masks
const __m128i mask1a = _mm_sub_epi16(*mask_max, *mask0a);
@@ -1187,11 +1187,11 @@
const __m128i *round_offset, int shift, const __m128i *clip_low,
const __m128i *clip_high, const __m128i *mask_max) {
do {
- const __m128i mask0a8 = _mm_set_epi32(0, 0, *(uint32_t *)mask,
- *(uint32_t *)(mask + mask_stride));
+ const __m128i mask0a8 =
+ _mm_set_epi32(0, 0, *(int32_t *)mask, *(int32_t *)(mask + mask_stride));
const __m128i mask0b8 =
- _mm_set_epi32(0, 0, *(uint32_t *)(mask + 2 * mask_stride),
- *(uint32_t *)(mask + 3 * mask_stride));
+ _mm_set_epi32(0, 0, *(int32_t *)(mask + 2 * mask_stride),
+ *(int32_t *)(mask + 3 * mask_stride));
const __m128i mask0a = _mm_cvtepu8_epi16(mask0a8);
const __m128i mask0b = _mm_cvtepu8_epi16(mask0b8);
@@ -1218,16 +1218,16 @@
// Load 8 pixels from each of 8 rows of mask,
// (saturating) add together rows then use madd to add adjacent pixels
// Finally, divide each value by 4 (with rounding)
- const __m128i m02 = _mm_set_epi64x(*(uint64_t *)(mask),
- *(uint64_t *)(mask + 2 * mask_stride));
- const __m128i m13 = _mm_set_epi64x(*(uint64_t *)(mask + mask_stride),
- *(uint64_t *)(mask + 3 * mask_stride));
+ const __m128i m02 = _mm_set_epi64x(*(int64_t *)(mask),
+ *(int64_t *)(mask + 2 * mask_stride));
+ const __m128i m13 = _mm_set_epi64x(*(int64_t *)(mask + mask_stride),
+ *(int64_t *)(mask + 3 * mask_stride));
const __m128i m0123 = _mm_maddubs_epi16(_mm_adds_epu8(m02, m13), one_b);
const __m128i mask_0a = _mm_srli_epi16(_mm_add_epi16(m0123, two_w), 2);
- const __m128i m46 = _mm_set_epi64x(*(uint64_t *)(mask + 4 * mask_stride),
- *(uint64_t *)(mask + 6 * mask_stride));
- const __m128i m57 = _mm_set_epi64x(*(uint64_t *)(mask + 5 * mask_stride),
- *(uint64_t *)(mask + 7 * mask_stride));
+ const __m128i m46 = _mm_set_epi64x(*(int64_t *)(mask + 4 * mask_stride),
+ *(int64_t *)(mask + 6 * mask_stride));
+ const __m128i m57 = _mm_set_epi64x(*(int64_t *)(mask + 5 * mask_stride),
+ *(int64_t *)(mask + 7 * mask_stride));
const __m128i m4567 = _mm_maddubs_epi16(_mm_adds_epu8(m46, m57), one_b);
const __m128i mask_0b = _mm_srli_epi16(_mm_add_epi16(m4567, two_w), 2);
@@ -1493,7 +1493,7 @@
const __m128i v_round_offset = _mm_set1_epi32(round_offset);
const int shift = round_bits + AOM_BLEND_A64_ROUND_BITS;
- const __m128i clip_low = _mm_set1_epi16(0);
+ const __m128i clip_low = _mm_setzero_si128();
const __m128i clip_high = _mm_set1_epi16((1 << bd) - 1);
const __m128i mask_max = _mm_set1_epi16(AOM_BLEND_A64_MAX_ALPHA);
diff --git a/aom_dsp/x86/fwd_txfm_sse2.h b/aom_dsp/x86/fwd_txfm_sse2.h
index ab3cd91..78ea985 100644
--- a/aom_dsp/x86/fwd_txfm_sse2.h
+++ b/aom_dsp/x86/fwd_txfm_sse2.h
@@ -34,7 +34,7 @@
static INLINE int check_epi16_overflow_x2(const __m128i *preg0,
const __m128i *preg1) {
const __m128i max_overflow = _mm_set1_epi16(0x7fff);
- const __m128i min_overflow = _mm_set1_epi16(0x8000);
+ const __m128i min_overflow = _mm_set1_epi16((short)0x8000);
__m128i cmp0 = _mm_or_si128(_mm_cmpeq_epi16(*preg0, max_overflow),
_mm_cmpeq_epi16(*preg0, min_overflow));
__m128i cmp1 = _mm_or_si128(_mm_cmpeq_epi16(*preg1, max_overflow),
@@ -48,7 +48,7 @@
const __m128i *preg2,
const __m128i *preg3) {
const __m128i max_overflow = _mm_set1_epi16(0x7fff);
- const __m128i min_overflow = _mm_set1_epi16(0x8000);
+ const __m128i min_overflow = _mm_set1_epi16((short)0x8000);
__m128i cmp0 = _mm_or_si128(_mm_cmpeq_epi16(*preg0, max_overflow),
_mm_cmpeq_epi16(*preg0, min_overflow));
__m128i cmp1 = _mm_or_si128(_mm_cmpeq_epi16(*preg1, max_overflow),
diff --git a/aom_dsp/x86/highbd_quantize_intrin_sse2.c b/aom_dsp/x86/highbd_quantize_intrin_sse2.c
index 1764a49..a5c450a 100644
--- a/aom_dsp/x86/highbd_quantize_intrin_sse2.c
+++ b/aom_dsp/x86/highbd_quantize_intrin_sse2.c
@@ -80,7 +80,8 @@
const int64_t tmp4 = ((tmp3 * quant_ptr[k != 0]) >> 16) + tmp3;
const uint32_t abs_qcoeff =
(uint32_t)((tmp4 * quant_shift_ptr[k != 0]) >> 16);
- qcoeff_ptr[k] = (int)(abs_qcoeff ^ coeff_sign[j]) - coeff_sign[j];
+ qcoeff_ptr[k] =
+ (int)(abs_qcoeff ^ (uint32_t)coeff_sign[j]) - coeff_sign[j];
dqcoeff_ptr[k] = qcoeff_ptr[k] * dequant_ptr[k != 0];
if (abs_qcoeff) eob_i = iscan[k] > eob_i ? iscan[k] : eob_i;
}
@@ -140,7 +141,7 @@
const int64_t tmp2 = ((tmp1 * quant_ptr[rc != 0]) >> 16) + tmp1;
const uint32_t abs_qcoeff =
(uint32_t)((tmp2 * quant_shift_ptr[rc != 0]) >> 15);
- qcoeff_ptr[rc] = (int)(abs_qcoeff ^ coeff_sign) - coeff_sign;
+ qcoeff_ptr[rc] = (int)(abs_qcoeff ^ (uint32_t)coeff_sign) - coeff_sign;
dqcoeff_ptr[rc] = qcoeff_ptr[rc] * dequant_ptr[rc != 0] / 2;
if (abs_qcoeff) eob = iscan[idx_arr[i]] > eob ? iscan[idx_arr[i]] : eob;
}
@@ -198,7 +199,7 @@
const int64_t tmp2 = ((tmp1 * quant_ptr[rc != 0]) >> 16) + tmp1;
const uint32_t abs_qcoeff =
(uint32_t)((tmp2 * quant_shift_ptr[rc != 0]) >> 14);
- qcoeff_ptr[rc] = (int)(abs_qcoeff ^ coeff_sign) - coeff_sign;
+ qcoeff_ptr[rc] = (int)(abs_qcoeff ^ (uint32_t)coeff_sign) - coeff_sign;
dqcoeff_ptr[rc] = qcoeff_ptr[rc] * dequant_ptr[rc != 0] / 4;
if (abs_qcoeff) eob = iscan[idx_arr[i]] > eob ? iscan[idx_arr[i]] : eob;
}
diff --git a/aom_dsp/x86/highbd_sad_avx2.c b/aom_dsp/x86/highbd_sad_avx2.c
index ad4db2f..f583772 100644
--- a/aom_dsp/x86/highbd_sad_avx2.c
+++ b/aom_dsp/x86/highbd_sad_avx2.c
@@ -551,7 +551,7 @@
static INLINE void get_4d_sad_from_mm256_epi32(const __m256i *v,
uint32_t *res) {
__m256i u0, u1, u2, u3;
- const __m256i mask = yy_set1_64_from_32i(UINT32_MAX);
+ const __m256i mask = yy_set1_64_from_32i(~0);
__m128i sad;
// 8 32-bit summation
diff --git a/aom_dsp/x86/highbd_variance_avx2.c b/aom_dsp/x86/highbd_variance_avx2.c
index 49912ac..36e6473 100644
--- a/aom_dsp/x86/highbd_variance_avx2.c
+++ b/aom_dsp/x86/highbd_variance_avx2.c
@@ -26,13 +26,13 @@
const uint32_t xoffset, const uint32_t yoffset, const uint8_t *dst_ptr8,
int dst_stride, uint32_t *sse) {
const __m256i filter1 =
- _mm256_set1_epi32((uint32_t)(bilinear_filters_2t[xoffset][1] << 16) |
+ _mm256_set1_epi32((int)(bilinear_filters_2t[xoffset][1] << 16) |
bilinear_filters_2t[xoffset][0]);
const __m256i filter2 =
- _mm256_set1_epi32((uint32_t)(bilinear_filters_2t[yoffset][1] << 16) |
+ _mm256_set1_epi32((int)(bilinear_filters_2t[yoffset][1] << 16) |
bilinear_filters_2t[yoffset][0]);
const __m256i one = _mm256_set1_epi16(1);
- const uint32_t bitshift = (uint32_t)0x40;
+ const int bitshift = 0x40;
(void)pixel_step;
unsigned int i, j, prev = 0, curr = 2;
uint16_t *src_ptr = CONVERT_TO_SHORTPTR(src_ptr8);
diff --git a/aom_dsp/x86/highbd_variance_sse2.c b/aom_dsp/x86/highbd_variance_sse2.c
index 6bd6a5a..d45885c 100644
--- a/aom_dsp/x86/highbd_variance_sse2.c
+++ b/aom_dsp/x86/highbd_variance_sse2.c
@@ -629,13 +629,12 @@
const uint8_t *ref8, int ref_stride,
const DIST_WTD_COMP_PARAMS *jcp_param) {
int i;
- const uint16_t wt0 = (uint16_t)jcp_param->fwd_offset;
- const uint16_t wt1 = (uint16_t)jcp_param->bck_offset;
- const __m128i w0 = _mm_set_epi16(wt0, wt0, wt0, wt0, wt0, wt0, wt0, wt0);
- const __m128i w1 = _mm_set_epi16(wt1, wt1, wt1, wt1, wt1, wt1, wt1, wt1);
- const uint16_t round = ((1 << DIST_PRECISION_BITS) >> 1);
- const __m128i r =
- _mm_set_epi16(round, round, round, round, round, round, round, round);
+ const int16_t wt0 = (int16_t)jcp_param->fwd_offset;
+ const int16_t wt1 = (int16_t)jcp_param->bck_offset;
+ const __m128i w0 = _mm_set1_epi16(wt0);
+ const __m128i w1 = _mm_set1_epi16(wt1);
+ const int16_t round = (int16_t)((1 << DIST_PRECISION_BITS) >> 1);
+ const __m128i r = _mm_set1_epi16(round);
uint16_t *pred = CONVERT_TO_SHORTPTR(pred8);
uint16_t *ref = CONVERT_TO_SHORTPTR(ref8);
uint16_t *comp_pred = CONVERT_TO_SHORTPTR(comp_pred8);
diff --git a/aom_dsp/x86/intrapred_avx2.c b/aom_dsp/x86/intrapred_avx2.c
index b5f7144..621ef7a 100644
--- a/aom_dsp/x86/intrapred_avx2.c
+++ b/aom_dsp/x86/intrapred_avx2.c
@@ -361,7 +361,7 @@
const uint8_t *left) {
(void)above;
(void)left;
- const __m256i row = _mm256_set1_epi8((uint8_t)0x80);
+ const __m256i row = _mm256_set1_epi8((int8_t)0x80);
row_store_32xh(&row, 32, dst, stride);
}
@@ -628,7 +628,7 @@
const uint8_t *left) {
(void)above;
(void)left;
- const __m256i row = _mm256_set1_epi8((uint8_t)0x80);
+ const __m256i row = _mm256_set1_epi8((int8_t)0x80);
row_store_32xh(&row, 16, dst, stride);
}
@@ -637,7 +637,7 @@
const uint8_t *left) {
(void)above;
(void)left;
- const __m256i row = _mm256_set1_epi8((uint8_t)0x80);
+ const __m256i row = _mm256_set1_epi8((int8_t)0x80);
row_store_32xh(&row, 64, dst, stride);
}
@@ -646,7 +646,7 @@
const uint8_t *left) {
(void)above;
(void)left;
- const __m256i row = _mm256_set1_epi8((uint8_t)0x80);
+ const __m256i row = _mm256_set1_epi8((int8_t)0x80);
row_store_64xh(&row, 64, dst, stride);
}
@@ -655,7 +655,7 @@
const uint8_t *left) {
(void)above;
(void)left;
- const __m256i row = _mm256_set1_epi8((uint8_t)0x80);
+ const __m256i row = _mm256_set1_epi8((int8_t)0x80);
row_store_64xh(&row, 32, dst, stride);
}
@@ -664,7 +664,7 @@
const uint8_t *left) {
(void)above;
(void)left;
- const __m256i row = _mm256_set1_epi8((uint8_t)0x80);
+ const __m256i row = _mm256_set1_epi8((int8_t)0x80);
row_store_64xh(&row, 16, dst, stride);
}
@@ -754,7 +754,7 @@
const uint8_t *above, const uint8_t *left) {
__m128i x = _mm_loadl_epi64((const __m128i *)left);
const __m256i l = _mm256_inserti128_si256(_mm256_castsi128_si256(x), x, 1);
- const __m256i tl16 = _mm256_set1_epi16((uint16_t)above[-1]);
+ const __m256i tl16 = _mm256_set1_epi16((int16_t)above[-1]);
__m256i rep = _mm256_set1_epi16((short)0x8000);
const __m256i one = _mm256_set1_epi16(1);
const __m256i top = get_top_vector(above);
@@ -778,7 +778,7 @@
void aom_paeth_predictor_16x16_avx2(uint8_t *dst, ptrdiff_t stride,
const uint8_t *above, const uint8_t *left) {
const __m256i l = get_left_vector(left);
- const __m256i tl16 = _mm256_set1_epi16((uint16_t)above[-1]);
+ const __m256i tl16 = _mm256_set1_epi16((int16_t)above[-1]);
__m256i rep = _mm256_set1_epi16((short)0x8000);
const __m256i one = _mm256_set1_epi16(1);
const __m256i top = get_top_vector(above);
@@ -797,7 +797,7 @@
void aom_paeth_predictor_16x32_avx2(uint8_t *dst, ptrdiff_t stride,
const uint8_t *above, const uint8_t *left) {
__m256i l = get_left_vector(left);
- const __m256i tl16 = _mm256_set1_epi16((uint16_t)above[-1]);
+ const __m256i tl16 = _mm256_set1_epi16((int16_t)above[-1]);
__m256i rep = _mm256_set1_epi16((short)0x8000);
const __m256i one = _mm256_set1_epi16(1);
const __m256i top = get_top_vector(above);
@@ -826,7 +826,7 @@
void aom_paeth_predictor_16x64_avx2(uint8_t *dst, ptrdiff_t stride,
const uint8_t *above, const uint8_t *left) {
- const __m256i tl16 = _mm256_set1_epi16((uint16_t)above[-1]);
+ const __m256i tl16 = _mm256_set1_epi16((int16_t)above[-1]);
const __m256i one = _mm256_set1_epi16(1);
const __m256i top = get_top_vector(above);
@@ -864,7 +864,7 @@
const __m256i l = get_left_vector(left);
const __m256i t0 = get_top_vector(above);
const __m256i t1 = get_top_vector(above + 16);
- const __m256i tl = _mm256_set1_epi16((uint16_t)above[-1]);
+ const __m256i tl = _mm256_set1_epi16((int16_t)above[-1]);
__m256i rep = _mm256_set1_epi16((short)0x8000);
const __m256i one = _mm256_set1_epi16(1);
@@ -886,7 +886,7 @@
__m256i l = get_left_vector(left);
const __m256i t0 = get_top_vector(above);
const __m256i t1 = get_top_vector(above + 16);
- const __m256i tl = _mm256_set1_epi16((uint16_t)above[-1]);
+ const __m256i tl = _mm256_set1_epi16((int16_t)above[-1]);
__m256i rep = _mm256_set1_epi16((short)0x8000);
const __m256i one = _mm256_set1_epi16(1);
@@ -924,7 +924,7 @@
const uint8_t *above, const uint8_t *left) {
const __m256i t0 = get_top_vector(above);
const __m256i t1 = get_top_vector(above + 16);
- const __m256i tl = _mm256_set1_epi16((uint16_t)above[-1]);
+ const __m256i tl = _mm256_set1_epi16((int16_t)above[-1]);
const __m256i one = _mm256_set1_epi16(1);
int i, j;
@@ -952,7 +952,7 @@
const __m256i t1 = get_top_vector(above + 16);
const __m256i t2 = get_top_vector(above + 32);
const __m256i t3 = get_top_vector(above + 48);
- const __m256i tl = _mm256_set1_epi16((uint16_t)above[-1]);
+ const __m256i tl = _mm256_set1_epi16((int16_t)above[-1]);
const __m256i one = _mm256_set1_epi16(1);
int i, j;
@@ -984,7 +984,7 @@
const __m256i t1 = get_top_vector(above + 16);
const __m256i t2 = get_top_vector(above + 32);
const __m256i t3 = get_top_vector(above + 48);
- const __m256i tl = _mm256_set1_epi16((uint16_t)above[-1]);
+ const __m256i tl = _mm256_set1_epi16((int16_t)above[-1]);
const __m256i one = _mm256_set1_epi16(1);
int i, j;
@@ -1016,7 +1016,7 @@
const __m256i t1 = get_top_vector(above + 16);
const __m256i t2 = get_top_vector(above + 32);
const __m256i t3 = get_top_vector(above + 48);
- const __m256i tl = _mm256_set1_epi16((uint16_t)above[-1]);
+ const __m256i tl = _mm256_set1_epi16((int16_t)above[-1]);
const __m256i one = _mm256_set1_epi16(1);
int i;
@@ -3537,7 +3537,7 @@
__m128i a_mbase_x;
a16 = _mm256_set1_epi16(16);
- a_mbase_x = _mm_set1_epi8(above[max_base_x]);
+ a_mbase_x = _mm_set1_epi8((int8_t)above[max_base_x]);
c3f = _mm256_set1_epi16(0x3f);
int x = dx;
@@ -3640,7 +3640,7 @@
__m256i a_mbase_x, diff, c3f;
a16 = _mm256_set1_epi16(16);
- a_mbase_x = _mm256_set1_epi8(above[max_base_x]);
+ a_mbase_x = _mm256_set1_epi8((int8_t)above[max_base_x]);
c3f = _mm256_set1_epi16(0x3f);
int x = dx;
@@ -3722,7 +3722,7 @@
__m128i max_base_x128, base_inc128, mask128;
a16 = _mm256_set1_epi16(16);
- a_mbase_x = _mm256_set1_epi8(above[max_base_x]);
+ a_mbase_x = _mm256_set1_epi8((int8_t)above[max_base_x]);
max_base_x128 = _mm_set1_epi8(max_base_x);
c3f = _mm256_set1_epi16(0x3f);
@@ -3766,14 +3766,14 @@
_mm256_extracti128_si256(res, 1))); // 16 8bit values
base_inc128 =
- _mm_setr_epi8((uint8_t)(base + j), (uint8_t)(base + j + 1),
- (uint8_t)(base + j + 2), (uint8_t)(base + j + 3),
- (uint8_t)(base + j + 4), (uint8_t)(base + j + 5),
- (uint8_t)(base + j + 6), (uint8_t)(base + j + 7),
- (uint8_t)(base + j + 8), (uint8_t)(base + j + 9),
- (uint8_t)(base + j + 10), (uint8_t)(base + j + 11),
- (uint8_t)(base + j + 12), (uint8_t)(base + j + 13),
- (uint8_t)(base + j + 14), (uint8_t)(base + j + 15));
+ _mm_setr_epi8((int8_t)(base + j), (int8_t)(base + j + 1),
+ (int8_t)(base + j + 2), (int8_t)(base + j + 3),
+ (int8_t)(base + j + 4), (int8_t)(base + j + 5),
+ (int8_t)(base + j + 6), (int8_t)(base + j + 7),
+ (int8_t)(base + j + 8), (int8_t)(base + j + 9),
+ (int8_t)(base + j + 10), (int8_t)(base + j + 11),
+ (int8_t)(base + j + 12), (int8_t)(base + j + 13),
+ (int8_t)(base + j + 14), (int8_t)(base + j + 15));
mask128 = _mm_cmpgt_epi8(_mm_subs_epu8(max_base_x128, base_inc128),
_mm_setzero_si128());
@@ -4092,7 +4092,7 @@
__m128i resx, resy;
__m128i resxy;
int y = r + 1;
- ydx = _mm256_set1_epi16((uint16_t)(y * dx));
+ ydx = _mm256_set1_epi16((int16_t)(y * dx));
int base_x = (-y * dx) >> frac_bits_x;
for (int j = 0; j < W; j += 16) {
diff --git a/aom_dsp/x86/intrapred_sse2.c b/aom_dsp/x86/intrapred_sse2.c
index 4786696..61e2973 100644
--- a/aom_dsp/x86/intrapred_sse2.c
+++ b/aom_dsp/x86/intrapred_sse2.c
@@ -146,7 +146,7 @@
sum += 6;
sum = divide_using_multiply_shift(sum, 2, DC_MULTIPLIER_1X2);
- const __m128i row = _mm_set1_epi8((uint8_t)sum);
+ const __m128i row = _mm_set1_epi8((int8_t)sum);
dc_store_8xh(&row, 4, dst, stride);
}
@@ -313,7 +313,7 @@
const uint8_t *above, const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_4(above);
- const __m128i two = _mm_set1_epi16((int16_t)2);
+ const __m128i two = _mm_set1_epi16(2);
sum_above = _mm_add_epi16(sum_above, two);
sum_above = _mm_srai_epi16(sum_above, 2);
sum_above = _mm_shufflelo_epi16(sum_above, 0);
@@ -327,7 +327,7 @@
const uint8_t *above, const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_4(above);
- const __m128i two = _mm_set1_epi16((int16_t)2);
+ const __m128i two = _mm_set1_epi16(2);
sum_above = _mm_add_epi16(sum_above, two);
sum_above = _mm_srai_epi16(sum_above, 2);
sum_above = _mm_shufflelo_epi16(sum_above, 0);
@@ -341,7 +341,7 @@
const uint8_t *above, const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_8(above);
- const __m128i four = _mm_set1_epi16((uint16_t)4);
+ const __m128i four = _mm_set1_epi16(4);
sum_above = _mm_add_epi16(sum_above, four);
sum_above = _mm_srai_epi16(sum_above, 3);
sum_above = _mm_unpacklo_epi8(sum_above, sum_above);
@@ -353,7 +353,7 @@
const uint8_t *above, const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_8(above);
- const __m128i four = _mm_set1_epi16((uint16_t)4);
+ const __m128i four = _mm_set1_epi16(4);
sum_above = _mm_add_epi16(sum_above, four);
sum_above = _mm_srai_epi16(sum_above, 3);
sum_above = _mm_unpacklo_epi8(sum_above, sum_above);
@@ -365,7 +365,7 @@
const uint8_t *above, const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_8(above);
- const __m128i four = _mm_set1_epi16((uint16_t)4);
+ const __m128i four = _mm_set1_epi16(4);
sum_above = _mm_add_epi16(sum_above, four);
sum_above = _mm_srai_epi16(sum_above, 3);
sum_above = _mm_unpacklo_epi8(sum_above, sum_above);
@@ -377,7 +377,7 @@
const uint8_t *above, const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_16_sse2(above);
- const __m128i eight = _mm_set1_epi16((uint16_t)8);
+ const __m128i eight = _mm_set1_epi16(8);
sum_above = _mm_add_epi16(sum_above, eight);
sum_above = _mm_srai_epi16(sum_above, 4);
sum_above = _mm_unpacklo_epi8(sum_above, sum_above);
@@ -390,7 +390,7 @@
const uint8_t *above, const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_16_sse2(above);
- const __m128i eight = _mm_set1_epi16((uint16_t)8);
+ const __m128i eight = _mm_set1_epi16(8);
sum_above = _mm_add_epi16(sum_above, eight);
sum_above = _mm_srai_epi16(sum_above, 4);
sum_above = _mm_unpacklo_epi8(sum_above, sum_above);
@@ -404,7 +404,7 @@
const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_16_sse2(above);
- const __m128i eight = _mm_set1_epi16((uint16_t)8);
+ const __m128i eight = _mm_set1_epi16(8);
sum_above = _mm_add_epi16(sum_above, eight);
sum_above = _mm_srai_epi16(sum_above, 4);
sum_above = _mm_unpacklo_epi8(sum_above, sum_above);
@@ -418,7 +418,7 @@
const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_16_sse2(above);
- const __m128i eight = _mm_set1_epi16((uint16_t)8);
+ const __m128i eight = _mm_set1_epi16(8);
sum_above = _mm_add_epi16(sum_above, eight);
sum_above = _mm_srai_epi16(sum_above, 4);
sum_above = _mm_unpacklo_epi8(sum_above, sum_above);
@@ -431,7 +431,7 @@
const uint8_t *above, const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_32_sse2(above);
- const __m128i sixteen = _mm_set1_epi16((uint16_t)16);
+ const __m128i sixteen = _mm_set1_epi16(16);
sum_above = _mm_add_epi16(sum_above, sixteen);
sum_above = _mm_srai_epi16(sum_above, 5);
sum_above = _mm_unpacklo_epi8(sum_above, sum_above);
@@ -445,7 +445,7 @@
const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_32_sse2(above);
- const __m128i sixteen = _mm_set1_epi16((uint16_t)16);
+ const __m128i sixteen = _mm_set1_epi16(16);
sum_above = _mm_add_epi16(sum_above, sixteen);
sum_above = _mm_srai_epi16(sum_above, 5);
sum_above = _mm_unpacklo_epi8(sum_above, sum_above);
@@ -459,7 +459,7 @@
const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_32_sse2(above);
- const __m128i sixteen = _mm_set1_epi16((uint16_t)16);
+ const __m128i sixteen = _mm_set1_epi16(16);
sum_above = _mm_add_epi16(sum_above, sixteen);
sum_above = _mm_srai_epi16(sum_above, 5);
sum_above = _mm_unpacklo_epi8(sum_above, sum_above);
@@ -473,7 +473,7 @@
const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_64(above);
- const __m128i thirtytwo = _mm_set1_epi16((uint16_t)32);
+ const __m128i thirtytwo = _mm_set1_epi16(32);
sum_above = _mm_add_epi16(sum_above, thirtytwo);
sum_above = _mm_srai_epi16(sum_above, 6);
sum_above = _mm_unpacklo_epi8(sum_above, sum_above);
@@ -487,7 +487,7 @@
const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_64(above);
- const __m128i thirtytwo = _mm_set1_epi16((uint16_t)32);
+ const __m128i thirtytwo = _mm_set1_epi16(32);
sum_above = _mm_add_epi16(sum_above, thirtytwo);
sum_above = _mm_srai_epi16(sum_above, 6);
sum_above = _mm_unpacklo_epi8(sum_above, sum_above);
@@ -501,7 +501,7 @@
const uint8_t *left) {
(void)left;
__m128i sum_above = dc_sum_64(above);
- const __m128i thirtytwo = _mm_set1_epi16((uint16_t)32);
+ const __m128i thirtytwo = _mm_set1_epi16(32);
sum_above = _mm_add_epi16(sum_above, thirtytwo);
sum_above = _mm_srai_epi16(sum_above, 6);
sum_above = _mm_unpacklo_epi8(sum_above, sum_above);
@@ -517,7 +517,7 @@
const uint8_t *above, const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_8(left);
- const __m128i four = _mm_set1_epi16((uint16_t)4);
+ const __m128i four = _mm_set1_epi16(4);
sum_left = _mm_add_epi16(sum_left, four);
sum_left = _mm_srai_epi16(sum_left, 3);
sum_left = _mm_shufflelo_epi16(sum_left, 0);
@@ -532,7 +532,7 @@
const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_16_sse2(left);
- const __m128i eight = _mm_set1_epi16((uint16_t)8);
+ const __m128i eight = _mm_set1_epi16(8);
sum_left = _mm_add_epi16(sum_left, eight);
sum_left = _mm_srai_epi16(sum_left, 4);
sum_left = _mm_shufflelo_epi16(sum_left, 0);
@@ -546,7 +546,7 @@
const uint8_t *above, const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_4(left);
- const __m128i two = _mm_set1_epi16((uint16_t)2);
+ const __m128i two = _mm_set1_epi16(2);
sum_left = _mm_add_epi16(sum_left, two);
sum_left = _mm_srai_epi16(sum_left, 2);
sum_left = _mm_unpacklo_epi8(sum_left, sum_left);
@@ -559,7 +559,7 @@
const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_16_sse2(left);
- const __m128i eight = _mm_set1_epi16((uint16_t)8);
+ const __m128i eight = _mm_set1_epi16(8);
sum_left = _mm_add_epi16(sum_left, eight);
sum_left = _mm_srai_epi16(sum_left, 4);
sum_left = _mm_unpacklo_epi8(sum_left, sum_left);
@@ -572,7 +572,7 @@
const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_32_sse2(left);
- const __m128i sixteen = _mm_set1_epi16((uint16_t)16);
+ const __m128i sixteen = _mm_set1_epi16(16);
sum_left = _mm_add_epi16(sum_left, sixteen);
sum_left = _mm_srai_epi16(sum_left, 5);
sum_left = _mm_unpacklo_epi8(sum_left, sum_left);
@@ -585,7 +585,7 @@
const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_4(left);
- const __m128i two = _mm_set1_epi16((uint16_t)2);
+ const __m128i two = _mm_set1_epi16(2);
sum_left = _mm_add_epi16(sum_left, two);
sum_left = _mm_srai_epi16(sum_left, 2);
sum_left = _mm_unpacklo_epi8(sum_left, sum_left);
@@ -599,7 +599,7 @@
const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_8(left);
- const __m128i four = _mm_set1_epi16((uint16_t)4);
+ const __m128i four = _mm_set1_epi16(4);
sum_left = _mm_add_epi16(sum_left, four);
sum_left = _mm_srai_epi16(sum_left, 3);
sum_left = _mm_unpacklo_epi8(sum_left, sum_left);
@@ -613,7 +613,7 @@
const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_32_sse2(left);
- const __m128i sixteen = _mm_set1_epi16((uint16_t)16);
+ const __m128i sixteen = _mm_set1_epi16(16);
sum_left = _mm_add_epi16(sum_left, sixteen);
sum_left = _mm_srai_epi16(sum_left, 5);
sum_left = _mm_unpacklo_epi8(sum_left, sum_left);
@@ -627,7 +627,7 @@
const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_64(left);
- const __m128i thirtytwo = _mm_set1_epi16((uint16_t)32);
+ const __m128i thirtytwo = _mm_set1_epi16(32);
sum_left = _mm_add_epi16(sum_left, thirtytwo);
sum_left = _mm_srai_epi16(sum_left, 6);
sum_left = _mm_unpacklo_epi8(sum_left, sum_left);
@@ -641,7 +641,7 @@
const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_8(left);
- const __m128i four = _mm_set1_epi16((uint16_t)4);
+ const __m128i four = _mm_set1_epi16(4);
sum_left = _mm_add_epi16(sum_left, four);
sum_left = _mm_srai_epi16(sum_left, 3);
sum_left = _mm_unpacklo_epi8(sum_left, sum_left);
@@ -655,7 +655,7 @@
const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_16_sse2(left);
- const __m128i eight = _mm_set1_epi16((uint16_t)8);
+ const __m128i eight = _mm_set1_epi16(8);
sum_left = _mm_add_epi16(sum_left, eight);
sum_left = _mm_srai_epi16(sum_left, 4);
sum_left = _mm_unpacklo_epi8(sum_left, sum_left);
@@ -669,7 +669,7 @@
const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_64(left);
- const __m128i thirtytwo = _mm_set1_epi16((uint16_t)32);
+ const __m128i thirtytwo = _mm_set1_epi16(32);
sum_left = _mm_add_epi16(sum_left, thirtytwo);
sum_left = _mm_srai_epi16(sum_left, 6);
sum_left = _mm_unpacklo_epi8(sum_left, sum_left);
@@ -683,7 +683,7 @@
const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_64(left);
- const __m128i thirtytwo = _mm_set1_epi16((uint16_t)32);
+ const __m128i thirtytwo = _mm_set1_epi16(32);
sum_left = _mm_add_epi16(sum_left, thirtytwo);
sum_left = _mm_srai_epi16(sum_left, 6);
sum_left = _mm_unpacklo_epi8(sum_left, sum_left);
@@ -697,7 +697,7 @@
const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_32_sse2(left);
- const __m128i sixteen = _mm_set1_epi16((uint16_t)16);
+ const __m128i sixteen = _mm_set1_epi16(16);
sum_left = _mm_add_epi16(sum_left, sixteen);
sum_left = _mm_srai_epi16(sum_left, 5);
sum_left = _mm_unpacklo_epi8(sum_left, sum_left);
@@ -711,7 +711,7 @@
const uint8_t *left) {
(void)above;
__m128i sum_left = dc_sum_16_sse2(left);
- const __m128i eight = _mm_set1_epi16((uint16_t)8);
+ const __m128i eight = _mm_set1_epi16(8);
sum_left = _mm_add_epi16(sum_left, eight);
sum_left = _mm_srai_epi16(sum_left, 4);
sum_left = _mm_unpacklo_epi8(sum_left, sum_left);
@@ -743,7 +743,7 @@
const uint8_t *above, const uint8_t *left) {
(void)above;
(void)left;
- const __m128i row = _mm_set1_epi8((uint8_t)128);
+ const __m128i row = _mm_set1_epi8((int8_t)128);
dc_store_8xh(&row, 4, dst, stride);
}
@@ -751,7 +751,7 @@
const uint8_t *above, const uint8_t *left) {
(void)above;
(void)left;
- const __m128i row = _mm_set1_epi8((uint8_t)128);
+ const __m128i row = _mm_set1_epi8((int8_t)128);
dc_store_8xh(&row, 16, dst, stride);
}
@@ -759,7 +759,7 @@
const uint8_t *above, const uint8_t *left) {
(void)above;
(void)left;
- const __m128i row = _mm_set1_epi8((uint8_t)128);
+ const __m128i row = _mm_set1_epi8((int8_t)128);
dc_store_8xh(&row, 32, dst, stride);
}
@@ -767,7 +767,7 @@
const uint8_t *above, const uint8_t *left) {
(void)above;
(void)left;
- const __m128i row = _mm_set1_epi8((uint8_t)128);
+ const __m128i row = _mm_set1_epi8((int8_t)128);
dc_store_16xh(&row, 4, dst, stride);
}
@@ -775,7 +775,7 @@
const uint8_t *above, const uint8_t *left) {
(void)above;
(void)left;
- const __m128i row = _mm_set1_epi8((uint8_t)128);
+ const __m128i row = _mm_set1_epi8((int8_t)128);
dc_store_16xh(&row, 8, dst, stride);
}
@@ -784,7 +784,7 @@
const uint8_t *left) {
(void)above;
(void)left;
- const __m128i row = _mm_set1_epi8((uint8_t)128);
+ const __m128i row = _mm_set1_epi8((int8_t)128);
dc_store_16xh(&row, 32, dst, stride);
}
@@ -793,7 +793,7 @@
const uint8_t *left) {
(void)above;
(void)left;
- const __m128i row = _mm_set1_epi8((uint8_t)128);
+ const __m128i row = _mm_set1_epi8((int8_t)128);
dc_store_16xh(&row, 64, dst, stride);
}
@@ -801,7 +801,7 @@
const uint8_t *above, const uint8_t *left) {
(void)above;
(void)left;
- const __m128i row = _mm_set1_epi8((uint8_t)128);
+ const __m128i row = _mm_set1_epi8((int8_t)128);
dc_store_32xh(&row, 8, dst, stride);
}
@@ -810,7 +810,7 @@
const uint8_t *left) {
(void)above;
(void)left;
- const __m128i row = _mm_set1_epi8((uint8_t)128);
+ const __m128i row = _mm_set1_epi8((int8_t)128);
dc_store_32xh(&row, 16, dst, stride);
}
@@ -819,7 +819,7 @@
const uint8_t *left) {
(void)above;
(void)left;
- const __m128i row = _mm_set1_epi8((uint8_t)128);
+ const __m128i row = _mm_set1_epi8((int8_t)128);
dc_store_32xh(&row, 64, dst, stride);
}
@@ -828,7 +828,7 @@
const uint8_t *left) {
(void)above;
(void)left;
- const __m128i row = _mm_set1_epi8((uint8_t)128);
+ const __m128i row = _mm_set1_epi8((int8_t)128);
dc_store_64xh(&row, 64, dst, stride);
}
@@ -837,7 +837,7 @@
const uint8_t *left) {
(void)above;
(void)left;
- const __m128i row = _mm_set1_epi8((uint8_t)128);
+ const __m128i row = _mm_set1_epi8((int8_t)128);
dc_store_64xh(&row, 32, dst, stride);
}
@@ -846,7 +846,7 @@
const uint8_t *left) {
(void)above;
(void)left;
- const __m128i row = _mm_set1_epi8((uint8_t)128);
+ const __m128i row = _mm_set1_epi8((int8_t)128);
dc_store_64xh(&row, 16, dst, stride);
}
@@ -1334,7 +1334,7 @@
const uint8_t *left, int height) {
int i = height >> 2;
do {
- __m128i left4 = _mm_cvtsi32_si128(((uint32_t *)left)[0]);
+ __m128i left4 = _mm_cvtsi32_si128(((int *)left)[0]);
left4 = _mm_unpacklo_epi8(left4, left4);
left4 = _mm_unpacklo_epi8(left4, left4);
const __m128i r0 = _mm_shuffle_epi32(left4, 0x0);
@@ -1364,7 +1364,7 @@
const uint8_t *left, int height) {
int i = height >> 2;
do {
- __m128i left4 = _mm_cvtsi32_si128(((uint32_t *)left)[0]);
+ __m128i left4 = _mm_cvtsi32_si128(((int *)left)[0]);
left4 = _mm_unpacklo_epi8(left4, left4);
left4 = _mm_unpacklo_epi8(left4, left4);
const __m128i r0 = _mm_shuffle_epi32(left4, 0x0);
diff --git a/aom_dsp/x86/intrapred_sse4.c b/aom_dsp/x86/intrapred_sse4.c
index 21fb1bb..3f72dc4 100644
--- a/aom_dsp/x86/intrapred_sse4.c
+++ b/aom_dsp/x86/intrapred_sse4.c
@@ -141,7 +141,7 @@
__m128i a_mbase_x;
a16 = _mm_set1_epi16(16);
- a_mbase_x = _mm_set1_epi8(above[max_base_x]);
+ a_mbase_x = _mm_set1_epi8((char)above[max_base_x]);
c3f = _mm_set1_epi16(0x3f);
int x = dx;
@@ -255,7 +255,7 @@
__m128i a_mbase_x, diff, c3f;
a16 = _mm_set1_epi16(16);
- a_mbase_x = _mm_set1_epi8(above[max_base_x]);
+ a_mbase_x = _mm_set1_epi8((char)above[max_base_x]);
c3f = _mm_set1_epi16(0x3f);
int x = dx;
@@ -353,7 +353,7 @@
__m128i max_base, base_inc, mask;
a16 = _mm_set1_epi16(16);
- a_mbase_x = _mm_set1_epi8(above[max_base_x]);
+ a_mbase_x = _mm_set1_epi8((char)above[max_base_x]);
max_base = _mm_set1_epi8(max_base_x);
c3f = _mm_set1_epi16(0x3f);
@@ -412,14 +412,14 @@
res = _mm_packus_epi16(res, res1); // 16 8bit values
base_inc =
- _mm_setr_epi8((uint8_t)(base + j), (uint8_t)(base + j + 1),
- (uint8_t)(base + j + 2), (uint8_t)(base + j + 3),
- (uint8_t)(base + j + 4), (uint8_t)(base + j + 5),
- (uint8_t)(base + j + 6), (uint8_t)(base + j + 7),
- (uint8_t)(base + j + 8), (uint8_t)(base + j + 9),
- (uint8_t)(base + j + 10), (uint8_t)(base + j + 11),
- (uint8_t)(base + j + 12), (uint8_t)(base + j + 13),
- (uint8_t)(base + j + 14), (uint8_t)(base + j + 15));
+ _mm_setr_epi8((int8_t)(base + j), (int8_t)(base + j + 1),
+ (int8_t)(base + j + 2), (int8_t)(base + j + 3),
+ (int8_t)(base + j + 4), (int8_t)(base + j + 5),
+ (int8_t)(base + j + 6), (int8_t)(base + j + 7),
+ (int8_t)(base + j + 8), (int8_t)(base + j + 9),
+ (int8_t)(base + j + 10), (int8_t)(base + j + 11),
+ (int8_t)(base + j + 12), (int8_t)(base + j + 13),
+ (int8_t)(base + j + 14), (int8_t)(base + j + 15));
mask = _mm_cmpgt_epi8(_mm_subs_epu8(max_base, base_inc),
_mm_setzero_si128());
@@ -743,7 +743,7 @@
__m128i resx, resy;
__m128i resxy;
int y = r + 1;
- ydx = _mm_set1_epi16((uint16_t)(y * dx));
+ ydx = _mm_set1_epi16((int16_t)(y * dx));
int base_x = (-y * dx) >> frac_bits_x;
for (int j = 0; j < W; j += 16) {
diff --git a/aom_dsp/x86/intrapred_ssse3.c b/aom_dsp/x86/intrapred_ssse3.c
index a75616e..ab59220 100644
--- a/aom_dsp/x86/intrapred_ssse3.c
+++ b/aom_dsp/x86/intrapred_ssse3.c
@@ -47,7 +47,7 @@
const __m128i t = _mm_loadl_epi64((const __m128i *)above);
const __m128i zero = _mm_setzero_si128();
const __m128i t16 = _mm_unpacklo_epi8(t, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
__m128i rep = _mm_set1_epi16((short)0x8000);
const __m128i one = _mm_set1_epi16(1);
@@ -68,7 +68,7 @@
const __m128i t = _mm_loadl_epi64((const __m128i *)above);
const __m128i zero = _mm_setzero_si128();
const __m128i t16 = _mm_unpacklo_epi8(t, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
__m128i rep = _mm_set1_epi16((short)0x8000);
const __m128i one = _mm_set1_epi16(1);
@@ -86,10 +86,10 @@
void aom_paeth_predictor_4x16_ssse3(uint8_t *dst, ptrdiff_t stride,
const uint8_t *above, const uint8_t *left) {
__m128i l = _mm_load_si128((const __m128i *)left);
- const __m128i t = _mm_cvtsi32_si128(((const uint32_t *)above)[0]);
+ const __m128i t = _mm_cvtsi32_si128(((const int *)above)[0]);
const __m128i zero = _mm_setzero_si128();
const __m128i t16 = _mm_unpacklo_epi8(t, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
__m128i rep = _mm_set1_epi16((short)0x8000);
const __m128i one = _mm_set1_epi16(1);
@@ -109,7 +109,7 @@
const __m128i t = _mm_loadl_epi64((const __m128i *)above);
const __m128i zero = _mm_setzero_si128();
const __m128i t16 = _mm_unpacklo_epi8(t, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
__m128i rep = _mm_set1_epi16((short)0x8000);
const __m128i one = _mm_set1_epi16(1);
@@ -130,7 +130,7 @@
const __m128i t = _mm_loadl_epi64((const __m128i *)above);
const __m128i zero = _mm_setzero_si128();
const __m128i t16 = _mm_unpacklo_epi8(t, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
__m128i rep = _mm_set1_epi16((short)0x8000);
const __m128i one = _mm_set1_epi16(1);
@@ -151,7 +151,7 @@
const __m128i t = _mm_loadl_epi64((const __m128i *)above);
const __m128i zero = _mm_setzero_si128();
const __m128i t16 = _mm_unpacklo_epi8(t, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
__m128i rep = _mm_set1_epi16((short)0x8000);
const __m128i one = _mm_set1_epi16(1);
@@ -171,7 +171,7 @@
const __m128i t = _mm_loadl_epi64((const __m128i *)above);
const __m128i zero = _mm_setzero_si128();
const __m128i t16 = _mm_unpacklo_epi8(t, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
const __m128i one = _mm_set1_epi16(1);
for (int j = 0; j < 2; ++j) {
@@ -199,12 +199,12 @@
void aom_paeth_predictor_16x4_ssse3(uint8_t *dst, ptrdiff_t stride,
const uint8_t *above, const uint8_t *left) {
- __m128i l = _mm_cvtsi32_si128(((const uint32_t *)left)[0]);
+ __m128i l = _mm_cvtsi32_si128(((const int *)left)[0]);
const __m128i t = _mm_load_si128((const __m128i *)above);
const __m128i zero = _mm_setzero_si128();
const __m128i top0 = _mm_unpacklo_epi8(t, zero);
const __m128i top1 = _mm_unpackhi_epi8(t, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
__m128i rep = _mm_set1_epi16((short)0x8000);
const __m128i one = _mm_set1_epi16(1);
@@ -225,7 +225,7 @@
const __m128i zero = _mm_setzero_si128();
const __m128i top0 = _mm_unpacklo_epi8(t, zero);
const __m128i top1 = _mm_unpackhi_epi8(t, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
__m128i rep = _mm_set1_epi16((short)0x8000);
const __m128i one = _mm_set1_epi16(1);
@@ -248,7 +248,7 @@
const __m128i zero = _mm_setzero_si128();
const __m128i top0 = _mm_unpacklo_epi8(t, zero);
const __m128i top1 = _mm_unpackhi_epi8(t, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
__m128i rep = _mm_set1_epi16((short)0x8000);
const __m128i one = _mm_set1_epi16(1);
@@ -271,7 +271,7 @@
const __m128i zero = _mm_setzero_si128();
const __m128i top0 = _mm_unpacklo_epi8(t, zero);
const __m128i top1 = _mm_unpackhi_epi8(t, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
__m128i rep = _mm_set1_epi16((short)0x8000);
const __m128i one = _mm_set1_epi16(1);
__m128i l16;
@@ -305,7 +305,7 @@
const __m128i zero = _mm_setzero_si128();
const __m128i top0 = _mm_unpacklo_epi8(t, zero);
const __m128i top1 = _mm_unpackhi_epi8(t, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
const __m128i one = _mm_set1_epi16(1);
for (int j = 0; j < 4; ++j) {
@@ -331,7 +331,7 @@
const __m128i bl = _mm_unpacklo_epi8(b, zero);
const __m128i bh = _mm_unpackhi_epi8(b, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
__m128i rep = _mm_set1_epi16((short)0x8000);
const __m128i one = _mm_set1_epi16(1);
const __m128i l = _mm_loadl_epi64((const __m128i *)left);
@@ -360,7 +360,7 @@
const __m128i bl = _mm_unpacklo_epi8(b, zero);
const __m128i bh = _mm_unpackhi_epi8(b, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
__m128i rep = _mm_set1_epi16((short)0x8000);
const __m128i one = _mm_set1_epi16(1);
__m128i l = _mm_load_si128((const __m128i *)left);
@@ -390,7 +390,7 @@
const __m128i bl = _mm_unpacklo_epi8(b, zero);
const __m128i bh = _mm_unpackhi_epi8(b, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
__m128i rep = _mm_set1_epi16((short)0x8000);
const __m128i one = _mm_set1_epi16(1);
__m128i l = _mm_load_si128((const __m128i *)left);
@@ -433,7 +433,7 @@
const __m128i bl = _mm_unpacklo_epi8(b, zero);
const __m128i bh = _mm_unpackhi_epi8(b, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
const __m128i one = _mm_set1_epi16(1);
__m128i l16;
@@ -471,7 +471,7 @@
const __m128i dl = _mm_unpacklo_epi8(d, zero);
const __m128i dh = _mm_unpackhi_epi8(d, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
const __m128i one = _mm_set1_epi16(1);
__m128i l16;
@@ -513,7 +513,7 @@
const __m128i dl = _mm_unpacklo_epi8(d, zero);
const __m128i dh = _mm_unpackhi_epi8(d, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
const __m128i one = _mm_set1_epi16(1);
__m128i l16;
@@ -555,7 +555,7 @@
const __m128i dl = _mm_unpacklo_epi8(d, zero);
const __m128i dh = _mm_unpackhi_epi8(d, zero);
- const __m128i tl16 = _mm_set1_epi16((uint16_t)above[-1]);
+ const __m128i tl16 = _mm_set1_epi16((int16_t)above[-1]);
const __m128i one = _mm_set1_epi16(1);
__m128i l16;
@@ -586,17 +586,17 @@
// pixels[2]: right_pred vector
static INLINE void load_pixel_w4(const uint8_t *above, const uint8_t *left,
int height, __m128i *pixels) {
- __m128i d = _mm_cvtsi32_si128(((const uint32_t *)above)[0]);
+ __m128i d = _mm_cvtsi32_si128(((const int *)above)[0]);
if (height == 4)
- pixels[1] = _mm_cvtsi32_si128(((const uint32_t *)left)[0]);
+ pixels[1] = _mm_cvtsi32_si128(((const int *)left)[0]);
else if (height == 8)
pixels[1] = _mm_loadl_epi64(((const __m128i *)left));
else
pixels[1] = _mm_loadu_si128(((const __m128i *)left));
- pixels[2] = _mm_set1_epi16((uint16_t)above[3]);
+ pixels[2] = _mm_set1_epi16((int16_t)above[3]);
- const __m128i bp = _mm_set1_epi16((uint16_t)left[height - 1]);
+ const __m128i bp = _mm_set1_epi16((int16_t)left[height - 1]);
const __m128i zero = _mm_setzero_si128();
d = _mm_unpacklo_epi8(d, zero);
pixels[0] = _mm_unpacklo_epi16(d, bp);
@@ -610,8 +610,8 @@
static INLINE void load_weight_w4(int height, __m128i *weight_h,
__m128i *weight_w) {
const __m128i zero = _mm_setzero_si128();
- const __m128i d = _mm_set1_epi16((uint16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
- const __m128i t = _mm_cvtsi32_si128(((const uint32_t *)smooth_weights)[0]);
+ const __m128i d = _mm_set1_epi16((int16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
+ const __m128i t = _mm_cvtsi32_si128(((const int *)smooth_weights)[0]);
weight_h[0] = _mm_unpacklo_epi8(t, zero);
weight_h[1] = _mm_sub_epi16(d, weight_h[0]);
weight_w[0] = _mm_unpacklo_epi16(weight_h[0], weight_h[1]);
@@ -711,16 +711,16 @@
static INLINE void load_pixel_w8(const uint8_t *above, const uint8_t *left,
int height, __m128i *pixels) {
const __m128i zero = _mm_setzero_si128();
- const __m128i bp = _mm_set1_epi16((uint16_t)left[height - 1]);
+ const __m128i bp = _mm_set1_epi16((int16_t)left[height - 1]);
__m128i d = _mm_loadl_epi64((const __m128i *)above);
d = _mm_unpacklo_epi8(d, zero);
pixels[0] = _mm_unpacklo_epi16(d, bp);
pixels[1] = _mm_unpackhi_epi16(d, bp);
- pixels[3] = _mm_set1_epi16((uint16_t)above[7]);
+ pixels[3] = _mm_set1_epi16((int16_t)above[7]);
if (height == 4) {
- pixels[2] = _mm_cvtsi32_si128(((const uint32_t *)left)[0]);
+ pixels[2] = _mm_cvtsi32_si128(((const int *)left)[0]);
} else if (height == 8) {
pixels[2] = _mm_loadl_epi64((const __m128i *)left);
} else if (height == 16) {
@@ -750,7 +750,7 @@
const int we_offset = height < 8 ? 0 : 4;
__m128i we = _mm_loadu_si128((const __m128i *)&smooth_weights[we_offset]);
weight_h[0] = _mm_unpacklo_epi8(we, zero);
- const __m128i d = _mm_set1_epi16((uint16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
+ const __m128i d = _mm_set1_epi16((int16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
weight_h[1] = _mm_sub_epi16(d, weight_h[0]);
if (height == 4) {
@@ -891,18 +891,18 @@
const uint8_t *const sm_weights_h = smooth_weights + bh - 4;
const __m128i zero = _mm_setzero_si128();
const __m128i scale_value =
- _mm_set1_epi16((uint16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
- const __m128i bottom_left = _mm_cvtsi32_si128((uint32_t)left[bh - 1]);
+ _mm_set1_epi16((int16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
+ const __m128i bottom_left = _mm_cvtsi32_si128(left[bh - 1]);
const __m128i dup16 = _mm_set1_epi32(0x01000100);
const __m128i top_right =
- _mm_shuffle_epi8(_mm_cvtsi32_si128((uint32_t)above[bw - 1]), dup16);
+ _mm_shuffle_epi8(_mm_cvtsi32_si128(above[bw - 1]), dup16);
const __m128i gat = _mm_set_epi32(0, 0, 0xe0c0a08, 0x6040200);
const __m128i round =
_mm_set1_epi32((uint16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
for (uint32_t y = 0; y < bh; ++y) {
- const __m128i weights_y = _mm_cvtsi32_si128((uint32_t)sm_weights_h[y]);
- const __m128i left_y = _mm_cvtsi32_si128((uint32_t)left[y]);
+ const __m128i weights_y = _mm_cvtsi32_si128(sm_weights_h[y]);
+ const __m128i left_y = _mm_cvtsi32_si128(left[y]);
const __m128i scale_m_weights_y = _mm_sub_epi16(scale_value, weights_y);
__m128i pred_scaled_bl = _mm_mullo_epi16(scale_m_weights_y, bottom_left);
const __m128i wl_y =
@@ -1023,8 +1023,8 @@
static INLINE void load_pixel_v_w4(const uint8_t *above, const uint8_t *left,
int height, __m128i *pixels) {
const __m128i zero = _mm_setzero_si128();
- __m128i d = _mm_cvtsi32_si128(((const uint32_t *)above)[0]);
- const __m128i bp = _mm_set1_epi16((uint16_t)left[height - 1]);
+ __m128i d = _mm_cvtsi32_si128(((const int *)above)[0]);
+ const __m128i bp = _mm_set1_epi16((int16_t)left[height - 1]);
d = _mm_unpacklo_epi8(d, zero);
pixels[0] = _mm_unpacklo_epi16(d, bp);
}
@@ -1033,11 +1033,10 @@
// weights[1]: scale - weights_h vector
static INLINE void load_weight_v_w4(int height, __m128i *weights) {
const __m128i zero = _mm_setzero_si128();
- const __m128i d = _mm_set1_epi16((uint16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
+ const __m128i d = _mm_set1_epi16((int16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
if (height == 4) {
- const __m128i weight =
- _mm_cvtsi32_si128(((const uint32_t *)smooth_weights)[0]);
+ const __m128i weight = _mm_cvtsi32_si128(((const int *)smooth_weights)[0]);
weights[0] = _mm_unpacklo_epi8(weight, zero);
weights[1] = _mm_sub_epi16(d, weights[0]);
} else if (height == 8) {
@@ -1121,7 +1120,7 @@
int height, __m128i *pixels) {
const __m128i zero = _mm_setzero_si128();
__m128i d = _mm_loadl_epi64((const __m128i *)above);
- const __m128i bp = _mm_set1_epi16((uint16_t)left[height - 1]);
+ const __m128i bp = _mm_set1_epi16((int16_t)left[height - 1]);
d = _mm_unpacklo_epi8(d, zero);
pixels[0] = _mm_unpacklo_epi16(d, bp);
pixels[1] = _mm_unpackhi_epi16(d, bp);
@@ -1137,7 +1136,7 @@
// weight_h[7]: same as [1], offset 24
static INLINE void load_weight_v_w8(int height, __m128i *weight_h) {
const __m128i zero = _mm_setzero_si128();
- const __m128i d = _mm_set1_epi16((uint16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
+ const __m128i d = _mm_set1_epi16((int16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
if (height < 16) {
const int offset = height < 8 ? 0 : 4;
@@ -1261,16 +1260,16 @@
const uint8_t *const sm_weights_h = smooth_weights + bh - 4;
const __m128i zero = _mm_setzero_si128();
const __m128i scale_value =
- _mm_set1_epi16((uint16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
+ _mm_set1_epi16((int16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
const __m128i dup16 = _mm_set1_epi32(0x01000100);
const __m128i bottom_left =
- _mm_shuffle_epi8(_mm_cvtsi32_si128((uint32_t)left[bh - 1]), dup16);
+ _mm_shuffle_epi8(_mm_cvtsi32_si128(left[bh - 1]), dup16);
const __m128i gat = _mm_set_epi32(0, 0, 0xe0c0a08, 0x6040200);
const __m128i round =
_mm_set1_epi32((uint16_t)(1 << (SMOOTH_WEIGHT_LOG2_SCALE - 1)));
for (uint32_t y = 0; y < bh; ++y) {
- const __m128i weights_y = _mm_cvtsi32_si128((uint32_t)sm_weights_h[y]);
+ const __m128i weights_y = _mm_cvtsi32_si128(sm_weights_h[y]);
const __m128i scale_m_weights_y =
_mm_shuffle_epi8(_mm_sub_epi16(scale_value, weights_y), dup16);
const __m128i wl_y =
@@ -1379,12 +1378,12 @@
static INLINE void load_pixel_h_w4(const uint8_t *above, const uint8_t *left,
int height, __m128i *pixels) {
if (height == 4)
- pixels[0] = _mm_cvtsi32_si128(((const uint32_t *)left)[0]);
+ pixels[0] = _mm_cvtsi32_si128(((const int *)left)[0]);
else if (height == 8)
pixels[0] = _mm_loadl_epi64(((const __m128i *)left));
else
pixels[0] = _mm_loadu_si128(((const __m128i *)left));
- pixels[1] = _mm_set1_epi16((uint16_t)above[3]);
+ pixels[1] = _mm_set1_epi16((int16_t)above[3]);
}
// weights[0]: weights_w and scale - weights_w interleave vector
@@ -1394,7 +1393,7 @@
const __m128i zero = _mm_setzero_si128();
const __m128i weights_0 = _mm_unpacklo_epi8(t, zero);
- const __m128i d = _mm_set1_epi16((uint16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
+ const __m128i d = _mm_set1_epi16((int16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
const __m128i weights_1 = _mm_sub_epi16(d, weights_0);
weights[0] = _mm_unpacklo_epi16(weights_0, weights_1);
}
@@ -1470,10 +1469,10 @@
// pixels[3]: right_pred vector
static INLINE void load_pixel_h_w8(const uint8_t *above, const uint8_t *left,
int height, __m128i *pixels) {
- pixels[1] = _mm_set1_epi16((uint16_t)above[7]);
+ pixels[1] = _mm_set1_epi16((int16_t)above[7]);
if (height == 4) {
- pixels[0] = _mm_cvtsi32_si128(((const uint32_t *)left)[0]);
+ pixels[0] = _mm_cvtsi32_si128(((const int *)left)[0]);
} else if (height == 8) {
pixels[0] = _mm_loadl_epi64((const __m128i *)left);
} else if (height == 16) {
@@ -1490,7 +1489,7 @@
static INLINE void load_weight_h_w8(int height, __m128i *weight_w) {
(void)height;
const __m128i zero = _mm_setzero_si128();
- const __m128i d = _mm_set1_epi16((uint16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
+ const __m128i d = _mm_set1_epi16((int16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
const __m128i we = _mm_loadu_si128((const __m128i *)&smooth_weights[4]);
const __m128i tmp1 = _mm_unpacklo_epi8(we, zero);
const __m128i tmp2 = _mm_sub_epi16(d, tmp1);
@@ -1592,14 +1591,14 @@
const uint8_t *const sm_weights_w = smooth_weights + bw - 4;
const __m128i zero = _mm_setzero_si128();
const __m128i scale_value =
- _mm_set1_epi16((uint16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
- const __m128i top_right = _mm_cvtsi32_si128((uint32_t)above[bw - 1]);
+ _mm_set1_epi16((int16_t)(1 << SMOOTH_WEIGHT_LOG2_SCALE));
+ const __m128i top_right = _mm_cvtsi32_si128(above[bw - 1]);
const __m128i gat = _mm_set_epi32(0, 0, 0xe0c0a08, 0x6040200);
const __m128i pred_round =
_mm_set1_epi32((1 << (SMOOTH_WEIGHT_LOG2_SCALE - 1)));
for (uint32_t y = 0; y < bh; ++y) {
- const __m128i left_y = _mm_cvtsi32_si128((uint32_t)left[y]);
+ const __m128i left_y = _mm_cvtsi32_si128(left[y]);
const __m128i tr_ly =
_mm_shuffle_epi32(_mm_unpacklo_epi16(top_right, left_y), 0);
diff --git a/aom_dsp/x86/jnt_variance_ssse3.c b/aom_dsp/x86/jnt_variance_ssse3.c
index 6ec5dd8..dd798ca 100644
--- a/aom_dsp/x86/jnt_variance_ssse3.c
+++ b/aom_dsp/x86/jnt_variance_ssse3.c
@@ -49,13 +49,12 @@
int ref_stride,
const DIST_WTD_COMP_PARAMS *jcp_param) {
int i;
- const uint8_t w0 = (uint8_t)jcp_param->fwd_offset;
- const uint8_t w1 = (uint8_t)jcp_param->bck_offset;
+ const int8_t w0 = (int8_t)jcp_param->fwd_offset;
+ const int8_t w1 = (int8_t)jcp_param->bck_offset;
const __m128i w = _mm_set_epi8(w1, w0, w1, w0, w1, w0, w1, w0, w1, w0, w1, w0,
w1, w0, w1, w0);
- const uint16_t round = ((1 << DIST_PRECISION_BITS) >> 1);
- const __m128i r =
- _mm_set_epi16(round, round, round, round, round, round, round, round);
+ const int16_t round = (int16_t)((1 << DIST_PRECISION_BITS) >> 1);
+ const __m128i r = _mm_set1_epi16(round);
if (width >= 16) {
// Read 16 pixels one row at a time
@@ -95,10 +94,10 @@
assert(!(width & 3));
assert(!(height & 3));
for (i = 0; i < height; i += 4) {
- const uint8_t *row0 = ref + 0 * ref_stride;
- const uint8_t *row1 = ref + 1 * ref_stride;
- const uint8_t *row2 = ref + 2 * ref_stride;
- const uint8_t *row3 = ref + 3 * ref_stride;
+ const int8_t *row0 = (const int8_t *)ref + 0 * ref_stride;
+ const int8_t *row1 = (const int8_t *)ref + 1 * ref_stride;
+ const int8_t *row2 = (const int8_t *)ref + 2 * ref_stride;
+ const int8_t *row3 = (const int8_t *)ref + 3 * ref_stride;
__m128i p0 =
_mm_setr_epi8(row0[0], row0[1], row0[2], row0[3], row1[0], row1[1],
diff --git a/aom_dsp/x86/loopfilter_avx2.c b/aom_dsp/x86/loopfilter_avx2.c
index b593819..af6c5da 100644
--- a/aom_dsp/x86/loopfilter_avx2.c
+++ b/aom_dsp/x86/loopfilter_avx2.c
@@ -32,7 +32,7 @@
_mm_broadcastb_epi8(_mm_cvtsi32_si128((int)_limit0[0]));
const __m128i blimit_v =
_mm_broadcastb_epi8(_mm_cvtsi32_si128((int)_blimit0[0]));
- const __m128i zero = _mm_set1_epi16(0);
+ const __m128i zero = _mm_setzero_si128();
const __m128i ff = _mm_cmpeq_epi8(zero, zero);
p256_2 =
@@ -239,7 +239,7 @@
_mm_broadcastb_epi8(_mm_cvtsi32_si128((int)_limit0[0]));
const __m128i blimit_v =
_mm_broadcastb_epi8(_mm_cvtsi32_si128((int)_blimit0[0]));
- const __m128i zero = _mm_set1_epi16(0);
+ const __m128i zero = _mm_setzero_si128();
const __m128i ff = _mm_cmpeq_epi8(zero, zero);
p256_3 =
@@ -486,7 +486,7 @@
_mm_broadcastb_epi8(_mm_cvtsi32_si128((int)_limit0[0]));
const __m128i blimit_v =
_mm_broadcastb_epi8(_mm_cvtsi32_si128((int)_blimit0[0]));
- const __m128i zero = _mm_set1_epi16(0);
+ const __m128i zero = _mm_setzero_si128();
const __m128i ff = _mm_cmpeq_epi8(zero, zero);
p256_3 =
diff --git a/aom_dsp/x86/loopfilter_sse2.c b/aom_dsp/x86/loopfilter_sse2.c
index 87c5bb3..731dd10 100644
--- a/aom_dsp/x86/loopfilter_sse2.c
+++ b/aom_dsp/x86/loopfilter_sse2.c
@@ -2133,7 +2133,7 @@
const unsigned char *_blimit0,
const unsigned char *_limit0,
const unsigned char *_thresh0) {
- const __m128i zero = _mm_set1_epi16(0);
+ const __m128i zero = _mm_setzero_si128();
const __m128i one = _mm_set1_epi8(1);
const __m128i blimit_v = _mm_load_si128((const __m128i *)_blimit0);
const __m128i limit_v = _mm_load_si128((const __m128i *)_limit0);
@@ -2438,7 +2438,7 @@
const unsigned char *_blimit0,
const unsigned char *_limit0,
const unsigned char *_thresh0) {
- const __m128i zero = _mm_set1_epi16(0);
+ const __m128i zero = _mm_setzero_si128();
const __m128i one = _mm_set1_epi8(1);
const __m128i blimit_v = _mm_load_si128((const __m128i *)_blimit0);
const __m128i limit_v = _mm_load_si128((const __m128i *)_limit0);
@@ -2630,7 +2630,7 @@
const unsigned char *_blimit0,
const unsigned char *_limit0,
const unsigned char *_thresh0) {
- const __m128i zero = _mm_set1_epi16(0);
+ const __m128i zero = _mm_setzero_si128();
const __m128i one = _mm_set1_epi8(1);
const __m128i blimit_v = _mm_load_si128((const __m128i *)_blimit0);
const __m128i limit_v = _mm_load_si128((const __m128i *)_limit0);
@@ -2802,7 +2802,7 @@
const unsigned char *_blimit0,
const unsigned char *_limit0,
const unsigned char *_thresh0) {
- const __m128i zero = _mm_set1_epi16(0);
+ const __m128i zero = _mm_setzero_si128();
const __m128i blimit_v = _mm_load_si128((const __m128i *)_blimit0);
const __m128i limit_v = _mm_load_si128((const __m128i *)_limit0);
const __m128i thresh_v = _mm_load_si128((const __m128i *)_thresh0);
diff --git a/aom_dsp/x86/masked_sad4d_ssse3.c b/aom_dsp/x86/masked_sad4d_ssse3.c
index 1235f27..799ce9e 100644
--- a/aom_dsp/x86/masked_sad4d_ssse3.c
+++ b/aom_dsp/x86/masked_sad4d_ssse3.c
@@ -153,15 +153,15 @@
_mm_storeu_si128((__m128i *)sad_array, res0);
}
-#define MASK_SAD4XH_ONE_REF(idx) \
- a = _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(uint32_t *)ref##idx), \
- _mm_cvtsi32_si128(*(uint32_t *)&ref##idx[a_stride])); \
- data = _mm_unpacklo_epi8(a, b); \
- mask = _mm_unpacklo_epi8(m, m_inv); \
- pred = _mm_maddubs_epi16(data, mask); \
- pred = xx_roundn_epu16(pred, AOM_BLEND_A64_ROUND_BITS); \
- \
- pred = _mm_packus_epi16(pred, _mm_setzero_si128()); \
+#define MASK_SAD4XH_ONE_REF(idx) \
+ a = _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(int *)ref##idx), \
+ _mm_cvtsi32_si128(*(int *)&ref##idx[a_stride])); \
+ data = _mm_unpacklo_epi8(a, b); \
+ mask = _mm_unpacklo_epi8(m, m_inv); \
+ pred = _mm_maddubs_epi16(data, mask); \
+ pred = xx_roundn_epu16(pred, AOM_BLEND_A64_ROUND_BITS); \
+ \
+ pred = _mm_packus_epi16(pred, _mm_setzero_si128()); \
res##idx = _mm_add_epi32(res##idx, _mm_sad_epu8(pred, src));
void aom_masked_sad4xhx4d_ssse3(const uint8_t *src_ptr, int src_stride,
@@ -182,15 +182,15 @@
const __m128i mask_max = _mm_set1_epi8((1 << AOM_BLEND_A64_ROUND_BITS));
for (int y = 0; y < height; y += 2) {
- const __m128i src = _mm_unpacklo_epi32(
- _mm_cvtsi32_si128(*(uint32_t *)src_ptr),
- _mm_cvtsi32_si128(*(uint32_t *)&src_ptr[src_stride]));
+ const __m128i src =
+ _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(int *)src_ptr),
+ _mm_cvtsi32_si128(*(int *)&src_ptr[src_stride]));
const __m128i b =
- _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(uint32_t *)b_ptr),
- _mm_cvtsi32_si128(*(uint32_t *)&b_ptr[b_stride]));
+ _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(int *)b_ptr),
+ _mm_cvtsi32_si128(*(int *)&b_ptr[b_stride]));
const __m128i m_copy =
- _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(uint32_t *)m_ptr),
- _mm_cvtsi32_si128(*(uint32_t *)&m_ptr[m_stride]));
+ _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(int *)m_ptr),
+ _mm_cvtsi32_si128(*(int *)&m_ptr[m_stride]));
__m128i m_inv = _mm_sub_epi8(mask_max, m_copy);
__m128i m = inv_mask ? m_inv : m_copy;
diff --git a/aom_dsp/x86/masked_sad_intrin_ssse3.c b/aom_dsp/x86/masked_sad_intrin_ssse3.c
index fd5352c..df3a876 100644
--- a/aom_dsp/x86/masked_sad_intrin_ssse3.c
+++ b/aom_dsp/x86/masked_sad_intrin_ssse3.c
@@ -194,18 +194,18 @@
for (y = 0; y < height; y += 2) {
// Load two rows at a time, this seems to be a bit faster
// than four rows at a time in this case.
- const __m128i src = _mm_unpacklo_epi32(
- _mm_cvtsi32_si128(*(uint32_t *)src_ptr),
- _mm_cvtsi32_si128(*(uint32_t *)&src_ptr[src_stride]));
+ const __m128i src =
+ _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(int *)src_ptr),
+ _mm_cvtsi32_si128(*(int *)&src_ptr[src_stride]));
const __m128i a =
- _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(uint32_t *)a_ptr),
- _mm_cvtsi32_si128(*(uint32_t *)&a_ptr[a_stride]));
+ _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(int *)a_ptr),
+ _mm_cvtsi32_si128(*(int *)&a_ptr[a_stride]));
const __m128i b =
- _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(uint32_t *)b_ptr),
- _mm_cvtsi32_si128(*(uint32_t *)&b_ptr[b_stride]));
+ _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(int *)b_ptr),
+ _mm_cvtsi32_si128(*(int *)&b_ptr[b_stride]));
const __m128i m =
- _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(uint32_t *)m_ptr),
- _mm_cvtsi32_si128(*(uint32_t *)&m_ptr[m_stride]));
+ _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(int *)m_ptr),
+ _mm_cvtsi32_si128(*(int *)&m_ptr[m_stride]));
const __m128i m_inv = _mm_sub_epi8(mask_max, m);
const __m128i data = _mm_unpacklo_epi8(a, b);
@@ -367,9 +367,8 @@
_mm_loadl_epi64((const __m128i *)&b_ptr[b_stride]));
// Zero-extend mask to 16 bits
const __m128i m = _mm_unpacklo_epi8(
- _mm_unpacklo_epi32(
- _mm_cvtsi32_si128(*(const uint32_t *)m_ptr),
- _mm_cvtsi32_si128(*(const uint32_t *)&m_ptr[m_stride])),
+ _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(const int *)m_ptr),
+ _mm_cvtsi32_si128(*(const int *)&m_ptr[m_stride])),
_mm_setzero_si128());
const __m128i m_inv = _mm_sub_epi16(mask_max, m);
diff --git a/aom_dsp/x86/masked_variance_intrin_ssse3.c b/aom_dsp/x86/masked_variance_intrin_ssse3.c
index ac0e576..6939aa4 100644
--- a/aom_dsp/x86/masked_variance_intrin_ssse3.c
+++ b/aom_dsp/x86/masked_variance_intrin_ssse3.c
@@ -494,15 +494,14 @@
for (y = 0; y < height; y += 4) {
// Load four rows at a time
- __m128i src =
- _mm_setr_epi32(*(uint32_t *)src_ptr, *(uint32_t *)&src_ptr[src_stride],
- *(uint32_t *)&src_ptr[src_stride * 2],
- *(uint32_t *)&src_ptr[src_stride * 3]);
+ __m128i src = _mm_setr_epi32(*(int *)src_ptr, *(int *)&src_ptr[src_stride],
+ *(int *)&src_ptr[src_stride * 2],
+ *(int *)&src_ptr[src_stride * 3]);
const __m128i a = _mm_loadu_si128((const __m128i *)a_ptr);
const __m128i b = _mm_loadu_si128((const __m128i *)b_ptr);
- const __m128i m = _mm_setr_epi32(
- *(uint32_t *)m_ptr, *(uint32_t *)&m_ptr[m_stride],
- *(uint32_t *)&m_ptr[m_stride * 2], *(uint32_t *)&m_ptr[m_stride * 3]);
+ const __m128i m = _mm_setr_epi32(*(int *)m_ptr, *(int *)&m_ptr[m_stride],
+ *(int *)&m_ptr[m_stride * 2],
+ *(int *)&m_ptr[m_stride * 3]);
accumulate_block(&src, &a, &b, &m, &sum, &sum_sq);
src_ptr += src_stride * 4;
@@ -986,9 +985,8 @@
const __m128i a = _mm_loadu_si128((const __m128i *)a_ptr);
const __m128i b = _mm_loadu_si128((const __m128i *)b_ptr);
const __m128i m = _mm_unpacklo_epi8(
- _mm_unpacklo_epi32(
- _mm_cvtsi32_si128(*(const uint32_t *)m_ptr),
- _mm_cvtsi32_si128(*(const uint32_t *)&m_ptr[m_stride])),
+ _mm_unpacklo_epi32(_mm_cvtsi32_si128(*(const int *)m_ptr),
+ _mm_cvtsi32_si128(*(const int *)&m_ptr[m_stride])),
zero);
const __m128i m_inv = _mm_sub_epi16(mask_max, m);
diff --git a/aom_dsp/x86/mem_sse2.h b/aom_dsp/x86/mem_sse2.h
index dacb613..085a572 100644
--- a/aom_dsp/x86/mem_sse2.h
+++ b/aom_dsp/x86/mem_sse2.h
@@ -19,20 +19,20 @@
#include "aom/aom_integer.h"
-static INLINE uint16_t loadu_uint16(const void *src) {
- uint16_t v;
+static INLINE int16_t loadu_int16(const void *src) {
+ int16_t v;
memcpy(&v, src, sizeof(v));
return v;
}
-static INLINE uint32_t loadu_uint32(const void *src) {
- uint32_t v;
+static INLINE int32_t loadu_int32(const void *src) {
+ int32_t v;
memcpy(&v, src, sizeof(v));
return v;
}
-static INLINE uint64_t loadu_uint64(const void *src) {
- uint64_t v;
+static INLINE int64_t loadu_int64(const void *src) {
+ int64_t v;
memcpy(&v, src, sizeof(v));
return v;
}
@@ -48,10 +48,10 @@
static INLINE __m128i load_8bit_4x4_to_1_reg_sse2(const void *const src,
const int byte_stride) {
- return _mm_setr_epi32(loadu_uint32((int8_t *)src + 0 * byte_stride),
- loadu_uint32((int8_t *)src + 1 * byte_stride),
- loadu_uint32((int8_t *)src + 2 * byte_stride),
- loadu_uint32((int8_t *)src + 3 * byte_stride));
+ return _mm_setr_epi32(loadu_int32((int8_t *)src + 0 * byte_stride),
+ loadu_int32((int8_t *)src + 1 * byte_stride),
+ loadu_int32((int8_t *)src + 2 * byte_stride),
+ loadu_int32((int8_t *)src + 3 * byte_stride));
}
static INLINE __m128i load_8bit_8x2_to_1_reg_sse2(const void *const src,
diff --git a/aom_dsp/x86/obmc_intrinsic_sse4.h b/aom_dsp/x86/obmc_intrinsic_sse4.h
index 5181e44..210f466 100644
--- a/aom_dsp/x86/obmc_intrinsic_sse4.h
+++ b/aom_dsp/x86/obmc_intrinsic_sse4.h
@@ -28,7 +28,7 @@
assert(IS_POWER_OF_TWO(h));
do {
- const __m128i v_p_b = _mm_cvtsi32_si128(*(const uint32_t *)(pre + n));
+ const __m128i v_p_b = _mm_cvtsi32_si128(*(const int *)(pre + n));
const __m128i v_m_d = _mm_load_si128((const __m128i *)(mask + n));
const __m128i v_w_d = _mm_load_si128((const __m128i *)(wsrc + n));
diff --git a/aom_dsp/x86/quantize_avx2.c b/aom_dsp/x86/quantize_avx2.c
index 5763bd6..b808d46 100644
--- a/aom_dsp/x86/quantize_avx2.c
+++ b/aom_dsp/x86/quantize_avx2.c
@@ -128,7 +128,7 @@
const int16_t *iscan) {
(void)scan;
__m256i v_zbin, v_round, v_quant, v_dequant, v_quant_shift;
- __m256i v_eobmax = _mm256_set1_epi16(0);
+ __m256i v_eobmax = _mm256_setzero_si256();
load_b_values_avx2(zbin_ptr, &v_zbin, round_ptr, &v_round, quant_ptr,
&v_quant, dequant_ptr, &v_dequant, quant_shift_ptr,
@@ -211,7 +211,7 @@
tran_low_t *dqcoeff_ptr, const int16_t *dequant_ptr, uint16_t *eob_ptr,
const int16_t *iscan, int log_scale) {
__m256i v_zbin, v_round, v_quant, v_dequant, v_quant_shift;
- __m256i v_eobmax = _mm256_set1_epi16(0);
+ __m256i v_eobmax = _mm256_setzero_si256();
load_b_values_avx2(zbin_ptr, &v_zbin, round_ptr, &v_round, quant_ptr,
&v_quant, dequant_ptr, &v_dequant, quant_shift_ptr,
diff --git a/aom_dsp/x86/sad4d_sse2.asm b/aom_dsp/x86/sad4d_sse2.asm
index 9ab44c1..6de708b 100644
--- a/aom_dsp/x86/sad4d_sse2.asm
+++ b/aom_dsp/x86/sad4d_sse2.asm
@@ -22,114 +22,95 @@
pavgb %2, m2
lea second_predq, [second_predq+8]
%endmacro
-; 'mflag' affect a lot how the code works.
+; 'spill_src_stride' affect a lot how the code works.
;
-; When 'mflag' is false, the 'src_strideq' resides in register,
-; [srcq + src_strideq + offset] is allowed, so we can simply
-; use such form to access src memory and don't bother to update
-; 'srcq' at each line. We only update 'srcq' each two-lines using
-; a compact LEA instruction like [srcq+src_strideq*2].
+; When 'spill_src_stride' is false, the 'src_strideq' resides in
+; register, [srcq + src_strideq + offset] is allowed, so we can simply
+; use such form to access src memory and don't bother to update 'srcq'
+; at each line. We only update 'srcq' each two-lines using a compact
+; LEA instruction like [srcq+src_strideq*2].
;
-; When 'mflag' is true, the 'src_strideq' resides in memory.
+; When 'spill_src_stride' is true, the 'src_strideq' resides in memory.
; we cannot use above form to access memory, we have to update
; 'srcq' at each line break. As we process two parts (first,second)
; together in each macro function, the second part may also sit
; in the next line, which means we also need to possibly add
; one 'src_strideq' to 'srcq' before processing second part.
-%macro HANDLE_FIRST_OFFSET 2
- %define first_offset %2
- %if mflag == 0 && %1 == 1
- %define first_offset (src_strideq + %2)
- %endif
-%endmacro
-
-; first_extraline, second_extraline, in_line_offset
-%macro HANDLE_SECOND_OFFSET 3
- %define second_offset %3
- %if mflag && %1 == 0 && %2 == 1
+%macro HANDLE_SECOND_OFFSET 0
+ %if spill_src_stride
+ %define second_offset 0
add srcq, src_strideq
- %endif
- %if mflag == 0 && %2 == 1
- %define second_offset (src_strideq + %3)
+ %else
+ %define second_offset (src_strideq)
%endif
%endmacro
-; Notes for line_ending:
-; 0 -- not a line ending
-; 1 -- line ending of a odd line [line numbers starts from one]
-; 2 -- line ending of a even line
; This is specically designed to handle when src_strideq is a
; memory position, under such case, we can not accomplish
; complex address calculation using LEA, and fall back to
; using simple ADD instruction at each line ending.
-%macro ADVANCE_END_OF_LINE 1
- %if mflag
+%macro ADVANCE_END_OF_TWO_LINES 0
+ %if spill_src_stride
add srcq, src_strideq
- %endif
- %if mflag == 0 && %1 == 2
- lea srcq, [srcq +src_strideq*2]
+ %else
+ lea srcq, [srcq+src_strideq*2]
%endif
- %if %1 == 2
- lea ref1q, [ref1q+ref_strideq*2]
- lea ref2q, [ref2q+ref_strideq*2]
- lea ref3q, [ref3q+ref_strideq*2]
- lea ref4q, [ref4q+ref_strideq*2]
- %endif
+; note: ref_stride is never spilled when processing two lines
+ lea ref1q, [ref1q+ref_strideq*2]
+ lea ref2q, [ref2q+ref_strideq*2]
+ lea ref3q, [ref3q+ref_strideq*2]
+ lea ref4q, [ref4q+ref_strideq*2]
%endmacro
-; Please note that the second_offset of src is for in_line_offset,
-; so it is less than src_stride.
-; PROCESS_4x2x4 first, off_{first,second}_{src,ref}, do_avg,
-; {first, second}_extraline, line_ending
-%macro PROCESS_4x2x4 9
- HANDLE_FIRST_OFFSET %7, %2
- movd m0, [srcq + first_offset]
- HANDLE_SECOND_OFFSET %7, %8, %4
+; PROCESS_4x2x4 first, do_avg
+%macro PROCESS_4x2x4 2
+ movd m0, [srcq]
+ HANDLE_SECOND_OFFSET
%if %1 == 1
- movd m6, [ref1q+%3]
- movd m4, [ref2q+%3]
- movd m7, [ref3q+%3]
- movd m5, [ref4q+%3]
+ movd m6, [ref1q]
+ movd m4, [ref2q]
+ movd m7, [ref3q]
+ movd m5, [ref4q]
movd m1, [srcq + second_offset]
- movd m2, [ref1q+%5]
+ movd m2, [ref1q+ref_strideq]
punpckldq m0, m1
punpckldq m6, m2
- movd m1, [ref2q+%5]
- movd m2, [ref3q+%5]
- movd m3, [ref4q+%5]
+ movd m1, [ref2q+ref_strideq]
+ movd m2, [ref3q+ref_strideq]
+ movd m3, [ref4q+ref_strideq]
punpckldq m4, m1
punpckldq m7, m2
punpckldq m5, m3
movlhps m0, m0
movlhps m6, m4
movlhps m7, m5
-%if %6 == 1
+%if %2 == 1
AVG_4x2x4 m6, m7
%endif
psadbw m6, m0
psadbw m7, m0
%else
- movd m1, [ref1q+%3]
- movd m5, [ref1q+%5]
- movd m2, [ref2q+%3]
- movd m4, [ref2q+%5]
+ movd m1, [ref1q]
+ movd m5, [ref1q+ref_strideq]
+ movd m2, [ref2q]
+ movd m4, [ref2q+ref_strideq]
punpckldq m1, m5
punpckldq m2, m4
- movd m3, [ref3q+%3]
- movd m5, [ref3q+%5]
+ movd m3, [ref3q]
+ movd m5, [ref3q+ref_strideq]
punpckldq m3, m5
- movd m4, [ref4q+%3]
- movd m5, [ref4q+%5]
+ movd m4, [ref4q]
+ movd m5, [ref4q+ref_strideq]
punpckldq m4, m5
movd m5, [srcq + second_offset]
punpckldq m0, m5
movlhps m0, m0
movlhps m1, m2
movlhps m3, m4
-%if %6 == 1
+%if %2 == 1
AVG_4x2x4 m1, m3
%endif
psadbw m1, m0
@@ -137,28 +118,23 @@
paddd m6, m1
paddd m7, m3
%endif
-%if %9 > 0
- ADVANCE_END_OF_LINE %9
-%endif
%endmacro
-; PROCESS_8x2x4 first, off_{first,second}_{src,ref}, do_avg,
-; {first,second}_extraline, line_ending
-%macro PROCESS_8x2x4 9
- HANDLE_FIRST_OFFSET %7, %2
- movh m0, [srcq + first_offset]
- HANDLE_SECOND_OFFSET %7, %8, %4
+; PROCESS_8x2x4 first, do_avg
+%macro PROCESS_8x2x4 2
+ movh m0, [srcq]
+ HANDLE_SECOND_OFFSET
%if %1 == 1
- movh m4, [ref1q+%3]
- movh m5, [ref2q+%3]
- movh m6, [ref3q+%3]
- movh m7, [ref4q+%3]
+ movh m4, [ref1q]
+ movh m5, [ref2q]
+ movh m6, [ref3q]
+ movh m7, [ref4q]
movhps m0, [srcq + second_offset]
- movhps m4, [ref1q+%5]
- movhps m5, [ref2q+%5]
- movhps m6, [ref3q+%5]
- movhps m7, [ref4q+%5]
-%if %6 == 1
+ movhps m4, [ref1q+ref_strideq]
+ movhps m5, [ref2q+ref_strideq]
+ movhps m6, [ref3q+ref_strideq]
+ movhps m7, [ref4q+ref_strideq]
+%if %2 == 1
movu m3, [second_predq]
pavgb m4, m3
pavgb m5, m3
@@ -171,12 +147,12 @@
psadbw m6, m0
psadbw m7, m0
%else
- movh m1, [ref1q+%3]
- movh m2, [ref2q+%3]
+ movh m1, [ref1q]
+ movh m2, [ref2q]
movhps m0, [srcq + second_offset]
- movhps m1, [ref1q+%5]
- movhps m2, [ref2q+%5]
-%if %6 == 1
+ movhps m1, [ref1q+ref_strideq]
+ movhps m2, [ref2q+ref_strideq]
+%if %2 == 1
movu m3, [second_predq]
pavgb m1, m3
pavgb m2, m3
@@ -186,11 +162,11 @@
paddd m4, m1
paddd m5, m2
- movh m1, [ref3q+%3]
- movhps m1, [ref3q+%5]
- movh m2, [ref4q+%3]
- movhps m2, [ref4q+%5]
-%if %6 == 1
+ movh m1, [ref3q]
+ movhps m1, [ref3q+ref_strideq]
+ movh m2, [ref4q]
+ movhps m2, [ref4q+ref_strideq]
+%if %2 == 1
pavgb m1, m3
pavgb m2, m3
lea second_predq, [second_predq+mmsize]
@@ -200,24 +176,16 @@
paddd m6, m1
paddd m7, m2
%endif
-%if %9 > 0
- ADVANCE_END_OF_LINE %9
-%endif
%endmacro
-; PROCESS_16x2x4 first, off_{first,second}_{src,ref}, do_avg,
-; {first,second}_extraline, line_ending
-%macro PROCESS_16x2x4 9
- ; 1st 16 px
- HANDLE_FIRST_OFFSET %7, %2
- mova m0, [srcq + first_offset]
- HANDLE_SECOND_OFFSET %7, %8, %4
+; PROCESS_FIRST_MMSIZE do_avg
+%macro PROCESS_FIRST_MMSIZE 1
+ mova m0, [srcq]
+ movu m4, [ref1q]
+ movu m5, [ref2q]
+ movu m6, [ref3q]
+ movu m7, [ref4q]
%if %1 == 1
- movu m4, [ref1q+%3]
- movu m5, [ref2q+%3]
- movu m6, [ref3q+%3]
- movu m7, [ref4q+%3]
-%if %6 == 1
movu m3, [second_predq]
pavgb m4, m3
pavgb m5, m3
@@ -229,10 +197,14 @@
psadbw m5, m0
psadbw m6, m0
psadbw m7, m0
-%else ; %1 == 1
- movu m1, [ref1q+%3]
- movu m2, [ref2q+%3]
-%if %6 == 1
+%endmacro
+
+; PROCESS_16x1x4 offset, do_avg
+%macro PROCESS_16x1x4 2
+ mova m0, [srcq + %1]
+ movu m1, [ref1q + ref_offsetq + %1]
+ movu m2, [ref2q + ref_offsetq + %1]
+%if %2 == 1
movu m3, [second_predq]
pavgb m1, m3
pavgb m2, m3
@@ -242,9 +214,9 @@
paddd m4, m1
paddd m5, m2
- movu m1, [ref3q+%3]
- movu m2, [ref4q+%3]
-%if %6 == 1
+ movu m1, [ref3q + ref_offsetq + %1]
+ movu m2, [ref4q + ref_offsetq + %1]
+%if %2 == 1
pavgb m1, m3
pavgb m2, m3
lea second_predq, [second_predq+mmsize]
@@ -253,60 +225,6 @@
psadbw m2, m0
paddd m6, m1
paddd m7, m2
-%endif ; %1 == 1
-
- ; 2nd 16 px
- mova m0, [srcq + second_offset]
- movu m1, [ref1q+%5]
- movu m2, [ref2q+%5]
-
-%if %6 == 1
- movu m3, [second_predq]
- pavgb m1, m3
- pavgb m2, m3
-%endif
- psadbw m1, m0
- psadbw m2, m0
- paddd m4, m1
- paddd m5, m2
-
- movu m1, [ref3q+%5]
- movu m2, [ref4q+%5]
-
-%if %9 > 0
- ADVANCE_END_OF_LINE %9
-%endif
-
-%if %6 == 1
- pavgb m1, m3
- pavgb m2, m3
- lea second_predq, [second_predq+mmsize]
-%endif
- psadbw m1, m0
- psadbw m2, m0
- paddd m6, m1
- paddd m7, m2
-%endmacro
-
-; PROCESS_32x2x4 first, off_{first,second}_{src,ref}, do_avg,
-; {first,second}_extraline, line_ending
-%macro PROCESS_32x2x4 9
- PROCESS_16x2x4 %1, %2, %3, %2 + 16, %3 + 16, %6, %7, %7, %8 - %7
- PROCESS_16x2x4 0, %4, %5, %4 + 16, %5 + 16, %6, %8, %8, %9
-%endmacro
-
-; PROCESS_64x2x4 first, off_{first,second}_{src,ref}, do_avg,
-; {first,second}_extraline, line_ending
-%macro PROCESS_64x2x4 9
- PROCESS_32x2x4 %1, %2, %3, %2 + 32, %3 + 32, %6, %7, %7, %8 - %7
- PROCESS_32x2x4 0, %4, %5, %4 + 32, %5 + 32, %6, %8, %8, %9
-%endmacro
-
-; PROCESS_128x2x4 first, off_{first,second}_{src,ref}, do_avg,
-; {first,second}_extraline, line_ending
-%macro PROCESS_128x2x4 9
- PROCESS_64x2x4 %1, %2, %3, %2 + 64, %3 + 64, %6, %7, %7, %8 - %7
- PROCESS_64x2x4 0, %4, %5, %4 + 64, %5 + 64, %6, %8, %8, %9
%endmacro
; void aom_sadNxNx4d_sse2(uint8_t *src, int src_stride,
@@ -318,38 +236,118 @@
; 3: If 0, then normal sad, else avg
; 4: If 0, then normal sad, else skip rows
%macro SADNXN4D 2-4 0,0
+
+%define spill_src_stride 0
+%define spill_ref_stride 0
+%define spill_cnt 0
+
+; Whether a shared offset should be used instead of adding strides to
+; each reference array. With this option, only one line will be processed
+; per loop iteration.
+%define use_ref_offset (%1 >= mmsize)
+
+; Remove loops in the 4x4 and 8x4 case
+%define use_loop (use_ref_offset || %2 > 4)
+
%if %4 == 1 ; skip rows
%if ARCH_X86_64
-cglobal sad_skip_%1x%2x4d, 5, 8, 8, src, src_stride, ref1, ref_stride, \
- res, ref2, ref3, ref4
+%if use_ref_offset
+cglobal sad_skip_%1x%2x4d, 5, 10, 8, src, src_stride, ref1, ref_stride, res, \
+ ref2, ref3, ref4, cnt, ref_offset
+%elif use_loop
+cglobal sad_skip_%1x%2x4d, 5, 9, 8, src, src_stride, ref1, ref_stride, res, \
+ ref2, ref3, ref4, cnt
%else
-cglobal sad_skip_%1x%2x4d, 4, 7, 8, src, src_stride, ref1, ref_stride, \
- ref2, ref3, ref4
+cglobal sad_skip_%1x%2x4d, 5, 8, 8, src, src_stride, ref1, ref_stride, res, \
+ ref2, ref3, ref4
+%endif
+%else
+%if use_ref_offset
+cglobal sad_skip_%1x%2x4d, 4, 7, 8, src, ref_offset, ref1, cnt, ref2, ref3, \
+ ref4
+%define spill_src_stride 1
+%define spill_ref_stride 1
+%elif use_loop
+cglobal sad_skip_%1x%2x4d, 4, 7, 8, src, cnt, ref1, ref_stride, ref2, \
+ ref3, ref4
+%define spill_src_stride 1
+%else
+cglobal sad_skip_%1x%2x4d, 4, 7, 8, src, src_stride, ref1, ref_stride, ref2, \
+ ref3, ref4
+%endif
%endif
%elif %3 == 0 ; normal sad
%if ARCH_X86_64
-cglobal sad%1x%2x4d, 5, 8, 8, src, src_stride, ref1, ref_stride, \
- res, ref2, ref3, ref4
+%if use_ref_offset
+cglobal sad%1x%2x4d, 5, 10, 8, src, src_stride, ref1, ref_stride, res, ref2, \
+ ref3, ref4, cnt, ref_offset
+%elif use_loop
+cglobal sad%1x%2x4d, 5, 9, 8, src, src_stride, ref1, ref_stride, res, ref2, \
+ ref3, ref4, cnt
%else
-cglobal sad%1x%2x4d, 4, 7, 8, src, src_stride, ref1, ref_stride, \
- ref2, ref3, ref4
+cglobal sad%1x%2x4d, 5, 8, 8, src, src_stride, ref1, ref_stride, res, ref2, \
+ ref3, ref4
+%endif
+%else
+%if use_ref_offset
+cglobal sad%1x%2x4d, 4, 7, 8, src, ref_offset, ref1, cnt, ref2, ref3, ref4
+ %define spill_src_stride 1
+ %define spill_ref_stride 1
+%elif use_loop
+cglobal sad%1x%2x4d, 4, 7, 8, src, cnt, ref1, ref_stride, ref2, ref3, ref4
+ %define spill_src_stride 1
+%else
+cglobal sad%1x%2x4d, 4, 7, 8, src, src_stride, ref1, ref_stride, ref2, ref3, \
+ ref4
+%endif
%endif
%else ; avg
%if ARCH_X86_64
+%if use_ref_offset
+cglobal sad%1x%2x4d_avg, 6, 11, 8, src, src_stride, ref1, ref_stride, \
+ second_pred, res, ref2, ref3, ref4, cnt, \
+ ref_offset
+%elif use_loop
cglobal sad%1x%2x4d_avg, 6, 10, 8, src, src_stride, ref1, ref_stride, \
- second_pred, res, ref2, ref3, ref4
+ second_pred, res, ref2, ref3, ref4, cnt
%else
-cglobal sad%1x%2x4d_avg, 5, 7, 8, src, ref4, ref1, ref_stride, \
- second_pred, ref2, ref3
- %define src_strideq r1mp
- %define src_strided r1mp
+cglobal sad%1x%2x4d_avg, 6, 9, 8, src, src_stride, ref1, ref_stride, \
+ second_pred, res, ref2, ref3, ref4
+%endif
+%else
+%if use_ref_offset
+cglobal sad%1x%2x4d_avg, 5, 7, 8, src, ref4, ref1, ref_offset, second_pred, ref2, ref3
+ %define spill_src_stride 1
+ %define spill_ref_stride 1
+ %define spill_cnt 1
+%elif use_loop
+cglobal sad%1x%2x4d_avg, 5, 7, 8, src, ref4, ref1, ref_stride, second_pred, ref2, ref3
+ %define spill_src_stride 1
+ %define spill_cnt 1
+%else
+cglobal sad%1x%2x4d_avg, 5, 7, 8, src, ref4, ref1, ref_stride, second_pred, ref2, ref3
+ %define spill_src_stride 1
+%endif
%endif
%endif
- %define mflag ((1 - ARCH_X86_64) & %3)
+%if spill_src_stride
+ %define src_strideq r1mp
+ %define src_strided r1mp
+%endif
+%if spill_ref_stride
+ %define ref_strideq r3mp
+ %define ref_strided r3mp
+%endif
+
+%if spill_cnt
+ SUB rsp, 4
+ %define cntd word [rsp]
+%endif
+
%if %4 == 1
- lea src_strided, [2*src_strided]
- lea ref_strided, [2*ref_strided]
+ sal src_strided, 1
+ sal ref_strided, 1
%endif
movsxdifnidn src_strideq, src_strided
movsxdifnidn ref_strideq, ref_strided
@@ -359,18 +357,67 @@
mov ref4q, [ref1q+gprsize*3]
mov ref1q, [ref1q+gprsize*0]
- PROCESS_%1x2x4 1, 0, 0, 0, ref_strideq, %3, 0, 1, 2
-%if %4 == 1 ; downsample number of rows by 2
-%define num_rep (%2-8)/4
-%else
-%define num_rep (%2-4)/2
-%endif
-%rep num_rep
- PROCESS_%1x2x4 0, 0, 0, 0, ref_strideq, %3, 0, 1, 2
-%endrep
-%undef num_rep
- PROCESS_%1x2x4 0, 0, 0, 0, ref_strideq, %3, 0, 1, 2
+; Is the loop for this wxh in another function?
+; If so, we jump into that function for the loop and returning
+%define external_loop (use_ref_offset && %1 > mmsize && %1 != %2)
+%if use_ref_offset
+ PROCESS_FIRST_MMSIZE %3
+%if %1 > mmsize
+ mov ref_offsetq, 0
+ mov cntd, %2 >> %4
+; Jump part way into the loop for the square version of this width
+%if %3 == 1
+ jmp mangle(private_prefix %+ _sad%1x%1x4d_avg %+ SUFFIX).midloop
+%elif %4 == 1
+ jmp mangle(private_prefix %+ _sad_skip_%1x%1x4d %+ SUFFIX).midloop
+%else
+ jmp mangle(private_prefix %+ _sad%1x%1x4d %+ SUFFIX).midloop
+%endif
+%else
+ mov ref_offsetq, ref_strideq
+ add srcq, src_strideq
+ mov cntd, (%2 >> %4) - 1
+%endif
+%if external_loop == 0
+.loop:
+; Unrolled horizontal loop
+%assign h_offset 0
+%rep %1/mmsize
+ PROCESS_16x1x4 h_offset, %3
+%if h_offset == 0
+; The first row of the first column is done outside the loop and jumps here
+.midloop:
+%endif
+%assign h_offset h_offset+mmsize
+%endrep
+
+ add srcq, src_strideq
+ add ref_offsetq, ref_strideq
+ sub cntd, 1
+ jnz .loop
+%endif
+%else
+ PROCESS_%1x2x4 1, %3
+ ADVANCE_END_OF_TWO_LINES
+%if use_loop
+ mov cntd, (%2/2 >> %4) - 1
+.loop:
+%endif
+ PROCESS_%1x2x4 0, %3
+%if use_loop
+ ADVANCE_END_OF_TWO_LINES
+ sub cntd, 1
+ jnz .loop
+%endif
+%endif
+
+%if spill_cnt
+; Undo stack allocation for cnt
+ ADD rsp, 4
+%endif
+
+%if external_loop == 0
%if %3 == 0
%define resultq r4
%define resultmp r4mp
@@ -379,6 +426,16 @@
%define resultmp r5mp
%endif
+; Undo modifications on parameters on the stack
+%if %4 == 1
+%if spill_src_stride
+ shr src_strided, 1
+%endif
+%if spill_ref_stride
+ shr ref_strided, 1
+%endif
+%endif
+
%if %1 > 4
pslldq m5, 4
pslldq m7, 4
@@ -407,6 +464,7 @@
movq [resultq+8], m7
RET
%endif
+%endif ; external_loop == 0
%endmacro
INIT_XMM sse2
diff --git a/aom_dsp/x86/sum_squares_avx2.c b/aom_dsp/x86/sum_squares_avx2.c
index 0d63db2..89b9b82 100644
--- a/aom_dsp/x86/sum_squares_avx2.c
+++ b/aom_dsp/x86/sum_squares_avx2.c
@@ -21,7 +21,7 @@
int width, int height) {
uint64_t result;
__m256i v_acc_q = _mm256_setzero_si256();
- const __m256i v_zext_mask_q = yy_set1_64_from_32i(0xffffffff);
+ const __m256i v_zext_mask_q = yy_set1_64_from_32i(~0);
for (int col = 0; col < height; col += 4) {
__m256i v_acc_d = _mm256_setzero_si256();
for (int row = 0; row < width; row += 16) {
diff --git a/aom_dsp/x86/sum_squares_sse2.c b/aom_dsp/x86/sum_squares_sse2.c
index 0bdeee9..25be856 100644
--- a/aom_dsp/x86/sum_squares_sse2.c
+++ b/aom_dsp/x86/sum_squares_sse2.c
@@ -84,7 +84,7 @@
src += stride << 2;
r += 4;
} while (r < height);
- const __m128i v_zext_mask_q = xx_set1_64_from_32i(0xffffffff);
+ const __m128i v_zext_mask_q = xx_set1_64_from_32i(~0);
__m128i v_acc_64 = _mm_add_epi64(_mm_srli_epi64(v_acc_q, 32),
_mm_and_si128(v_acc_q, v_zext_mask_q));
v_acc_64 = _mm_add_epi64(v_acc_64, _mm_srli_si128(v_acc_64, 8));
@@ -116,7 +116,7 @@
int height) {
int r = 0;
- const __m128i v_zext_mask_q = xx_set1_64_from_32i(0xffffffff);
+ const __m128i v_zext_mask_q = xx_set1_64_from_32i(~0);
__m128i v_acc_q = _mm_setzero_si128();
do {
@@ -254,7 +254,7 @@
//////////////////////////////////////////////////////////////////////////////
static uint64_t aom_sum_squares_i16_64n_sse2(const int16_t *src, uint32_t n) {
- const __m128i v_zext_mask_q = xx_set1_64_from_32i(0xffffffff);
+ const __m128i v_zext_mask_q = xx_set1_64_from_32i(~0);
__m128i v_acc0_q = _mm_setzero_si128();
__m128i v_acc1_q = _mm_setzero_si128();
@@ -306,7 +306,7 @@
if (n % 64 == 0) {
return aom_sum_squares_i16_64n_sse2(src, n);
} else if (n > 64) {
- int k = n & ~(64 - 1);
+ const uint32_t k = n & ~63u;
return aom_sum_squares_i16_64n_sse2(src, k) +
aom_sum_squares_i16_c(src + k, n - k);
} else {
diff --git a/aom_dsp/x86/variance_avx2.c b/aom_dsp/x86/variance_avx2.c
index 7398a73..a7203ec 100644
--- a/aom_dsp/x86/variance_avx2.c
+++ b/aom_dsp/x86/variance_avx2.c
@@ -234,19 +234,20 @@
return *sse;
}
-unsigned int aom_sub_pixel_variance32xh_avx2(const uint8_t *src, int src_stride,
- int x_offset, int y_offset,
- const uint8_t *dst, int dst_stride,
- int height, unsigned int *sse);
-unsigned int aom_sub_pixel_variance16xh_avx2(const uint8_t *src, int src_stride,
- int x_offset, int y_offset,
- const uint8_t *dst, int dst_stride,
- int height, unsigned int *sse);
+int aom_sub_pixel_variance32xh_avx2(const uint8_t *src, int src_stride,
+ int x_offset, int y_offset,
+ const uint8_t *dst, int dst_stride,
+ int height, unsigned int *sse);
+int aom_sub_pixel_variance16xh_avx2(const uint8_t *src, int src_stride,
+ int x_offset, int y_offset,
+ const uint8_t *dst, int dst_stride,
+ int height, unsigned int *sse);
-unsigned int aom_sub_pixel_avg_variance32xh_avx2(
- const uint8_t *src, int src_stride, int x_offset, int y_offset,
- const uint8_t *dst, int dst_stride, const uint8_t *sec, int sec_stride,
- int height, unsigned int *sseptr);
+int aom_sub_pixel_avg_variance32xh_avx2(const uint8_t *src, int src_stride,
+ int x_offset, int y_offset,
+ const uint8_t *dst, int dst_stride,
+ const uint8_t *sec, int sec_stride,
+ int height, unsigned int *sseptr);
#define AOM_SUB_PIXEL_VAR_AVX2(w, h, wf, wlog2, hlog2) \
unsigned int aom_sub_pixel_variance##w##x##h##_avx2( \
@@ -539,10 +540,10 @@
const __m256i zeros = _mm256_broadcastsi128_si256(_mm_setzero_si128());
__m256i square_result = _mm256_broadcastsi128_si256(_mm_setzero_si128());
for (int i = 0; i < h; i += 4) {
- dst0_4x8 = _mm_cvtsi32_si128(*(uint32_t const *)(&dst[(i + 0) * dstride]));
- dst1_4x8 = _mm_cvtsi32_si128(*(uint32_t const *)(&dst[(i + 1) * dstride]));
- dst2_4x8 = _mm_cvtsi32_si128(*(uint32_t const *)(&dst[(i + 2) * dstride]));
- dst3_4x8 = _mm_cvtsi32_si128(*(uint32_t const *)(&dst[(i + 3) * dstride]));
+ dst0_4x8 = _mm_cvtsi32_si128(*(int const *)(&dst[(i + 0) * dstride]));
+ dst1_4x8 = _mm_cvtsi32_si128(*(int const *)(&dst[(i + 1) * dstride]));
+ dst2_4x8 = _mm_cvtsi32_si128(*(int const *)(&dst[(i + 2) * dstride]));
+ dst3_4x8 = _mm_cvtsi32_si128(*(int const *)(&dst[(i + 3) * dstride]));
dst_16x8 = _mm_unpacklo_epi64(_mm_unpacklo_epi32(dst0_4x8, dst1_4x8),
_mm_unpacklo_epi32(dst2_4x8, dst3_4x8));
dst_16x16 = _mm256_cvtepu8_epi16(dst_16x8);
diff --git a/aom_dsp/x86/variance_impl_avx2.c b/aom_dsp/x86/variance_impl_avx2.c
index 163e4cc..8ea0443 100644
--- a/aom_dsp/x86/variance_impl_avx2.c
+++ b/aom_dsp/x86/variance_impl_avx2.c
@@ -163,17 +163,17 @@
src_lo = _mm_srai_epi16(src_lo, 4); \
src_hi = _mm_srai_epi16(src_hi, 4);
-unsigned int aom_sub_pixel_variance32xh_avx2(const uint8_t *src, int src_stride,
- int x_offset, int y_offset,
- const uint8_t *dst, int dst_stride,
- int height, unsigned int *sse) {
+int aom_sub_pixel_variance32xh_avx2(const uint8_t *src, int src_stride,
+ int x_offset, int y_offset,
+ const uint8_t *dst, int dst_stride,
+ int height, unsigned int *sse) {
__m256i src_reg, dst_reg, exp_src_lo, exp_src_hi, exp_dst_lo, exp_dst_hi;
__m256i sse_reg, sum_reg, sse_reg_hi, res_cmp, sum_reg_lo, sum_reg_hi;
__m256i zero_reg;
int i, sum;
- sum_reg = _mm256_set1_epi16(0);
- sse_reg = _mm256_set1_epi16(0);
- zero_reg = _mm256_set1_epi16(0);
+ sum_reg = _mm256_setzero_si256();
+ sse_reg = _mm256_setzero_si256();
+ zero_reg = _mm256_setzero_si256();
// x_offset = 0 and y_offset = 0
if (x_offset == 0) {
@@ -351,17 +351,17 @@
return sum;
}
-unsigned int aom_sub_pixel_variance16xh_avx2(const uint8_t *src, int src_stride,
- int x_offset, int y_offset,
- const uint8_t *dst, int dst_stride,
- int height, unsigned int *sse) {
+int aom_sub_pixel_variance16xh_avx2(const uint8_t *src, int src_stride,
+ int x_offset, int y_offset,
+ const uint8_t *dst, int dst_stride,
+ int height, unsigned int *sse) {
__m256i src_reg, dst_reg, exp_src_lo, exp_src_hi, exp_dst_lo, exp_dst_hi;
__m256i sse_reg, sum_reg, sse_reg_hi, res_cmp, sum_reg_lo, sum_reg_hi;
__m256i zero_reg;
int i, sum;
- sum_reg = _mm256_set1_epi16(0);
- sse_reg = _mm256_set1_epi16(0);
- zero_reg = _mm256_set1_epi16(0);
+ sum_reg = _mm256_setzero_si256();
+ sse_reg = _mm256_setzero_si256();
+ zero_reg = _mm256_setzero_si256();
// x_offset = 0 and y_offset = 0
if (x_offset == 0) {
@@ -589,18 +589,19 @@
return sum;
}
-unsigned int aom_sub_pixel_avg_variance32xh_avx2(
- const uint8_t *src, int src_stride, int x_offset, int y_offset,
- const uint8_t *dst, int dst_stride, const uint8_t *sec, int sec_stride,
- int height, unsigned int *sse) {
+int aom_sub_pixel_avg_variance32xh_avx2(const uint8_t *src, int src_stride,
+ int x_offset, int y_offset,
+ const uint8_t *dst, int dst_stride,
+ const uint8_t *sec, int sec_stride,
+ int height, unsigned int *sse) {
__m256i sec_reg;
__m256i src_reg, dst_reg, exp_src_lo, exp_src_hi, exp_dst_lo, exp_dst_hi;
__m256i sse_reg, sum_reg, sse_reg_hi, res_cmp, sum_reg_lo, sum_reg_hi;
__m256i zero_reg;
int i, sum;
- sum_reg = _mm256_set1_epi16(0);
- sse_reg = _mm256_set1_epi16(0);
- zero_reg = _mm256_set1_epi16(0);
+ sum_reg = _mm256_setzero_si256();
+ sse_reg = _mm256_setzero_si256();
+ zero_reg = _mm256_setzero_si256();
// x_offset = 0 and y_offset = 0
if (x_offset == 0) {
diff --git a/aom_dsp/x86/variance_impl_ssse3.c b/aom_dsp/x86/variance_impl_ssse3.c
index 66b0d7d..6990021 100644
--- a/aom_dsp/x86/variance_impl_ssse3.c
+++ b/aom_dsp/x86/variance_impl_ssse3.c
@@ -25,8 +25,8 @@
// Change {128, 0} to {64, 0} and reduce FILTER_BITS by 1 to avoid overflow.
const int16_t round = (1 << (FILTER_BITS - 1)) >> 1;
const __m128i r = _mm_set1_epi16(round);
- const uint8_t f0 = filter[0] >> 1;
- const uint8_t f1 = filter[1] >> 1;
+ const int8_t f0 = (int8_t)(filter[0] >> 1);
+ const int8_t f1 = (int8_t)(filter[1] >> 1);
const __m128i filters = _mm_setr_epi8(f0, f1, f0, f1, f0, f1, f0, f1, f0, f1,
f0, f1, f0, f1, f0, f1);
unsigned int i, j;
diff --git a/aom_dsp/x86/variance_sse2.c b/aom_dsp/x86/variance_sse2.c
index a0223a9..6fb596d 100644
--- a/aom_dsp/x86/variance_sse2.c
+++ b/aom_dsp/x86/variance_sse2.c
@@ -36,8 +36,8 @@
}
static INLINE __m128i load4x2_sse2(const uint8_t *const p, const int stride) {
- const __m128i p0 = _mm_cvtsi32_si128(loadu_uint32(p + 0 * stride));
- const __m128i p1 = _mm_cvtsi32_si128(loadu_uint32(p + 1 * stride));
+ const __m128i p0 = _mm_cvtsi32_si128(loadu_int32(p + 0 * stride));
+ const __m128i p1 = _mm_cvtsi32_si128(loadu_int32(p + 1 * stride));
return _mm_unpacklo_epi8(_mm_unpacklo_epi32(p0, p1), _mm_setzero_si128());
}
@@ -103,7 +103,7 @@
vsum = _mm_add_epi16(vsum, _mm_srli_si128(vsum, 8));
vsum = _mm_unpacklo_epi16(vsum, vsum);
vsum = _mm_srai_epi32(vsum, 16);
- *sum = add32x4_sse2(vsum);
+ *sum = (int)add32x4_sse2(vsum);
}
// Can handle 1024 pixels' diff sum (such as 32x32)
@@ -113,7 +113,7 @@
*sse = add32x4_sse2(vsse);
vsum = sum_to_32bit_sse2(vsum);
- *sum = add32x4_sse2(vsum);
+ *sum = (int)add32x4_sse2(vsum);
}
static INLINE void variance4_sse2(const uint8_t *src, const int src_stride,
@@ -314,7 +314,7 @@
ref += (ref_stride * uh); \
} \
*sse = add32x4_sse2(vsse); \
- int sum = add32x4_sse2(vsum); \
+ int sum = (int)add32x4_sse2(vsum); \
assert(sum <= 255 * bw * bh); \
assert(sum >= -255 * bw * bh); \
return *sse - (uint32_t)(((int64_t)sum * sum) >> bits); \
@@ -678,8 +678,8 @@
const __m128i zeros = _mm_setzero_si128();
__m128i square_result = _mm_setzero_si128();
for (int i = 0; i < h; i += 2) {
- dst0_8x8 = _mm_cvtsi32_si128(*(uint32_t const *)(&dst[(i + 0) * dstride]));
- dst1_8x8 = _mm_cvtsi32_si128(*(uint32_t const *)(&dst[(i + 1) * dstride]));
+ dst0_8x8 = _mm_cvtsi32_si128(*(int const *)(&dst[(i + 0) * dstride]));
+ dst1_8x8 = _mm_cvtsi32_si128(*(int const *)(&dst[(i + 1) * dstride]));
dst_16x8 = _mm_unpacklo_epi8(_mm_unpacklo_epi32(dst0_8x8, dst1_8x8), zeros);
src0_16x4 = _mm_loadl_epi64((__m128i const *)(&src[(i + 0) * sstride]));
diff --git a/aom_ports/mem.h b/aom_ports/mem.h
index e9bb8ad..e396842 100644
--- a/aom_ports/mem.h
+++ b/aom_ports/mem.h
@@ -71,6 +71,8 @@
#define ALIGN_POWER_OF_TWO(value, n) \
(((value) + ((1 << (n)) - 1)) & ~((1 << (n)) - 1))
+#define ALIGN_POWER_OF_TWO_UNSIGNED(value, n) \
+ (((value) + ((1u << (n)) - 1)) & ~((1u << (n)) - 1))
#define DIVIDE_AND_ROUND(x, y) (((x) + ((y) >> 1)) / (y))
diff --git a/aom_ports/x86.h b/aom_ports/x86.h
index 79cbd02..f390dfa 100644
--- a/aom_ports/x86.h
+++ b/aom_ports/x86.h
@@ -387,7 +387,7 @@
// Reserved 01B
// Double Precision (53-Bits) 10B
// Extended Precision (64-Bits) 11B
- x87_set_control_word((mode & ~0x300) | 0x200);
+ x87_set_control_word((mode & ~0x300u) | 0x200u);
return mode;
}
diff --git a/aom_scale/aom_scale_rtcd.pl b/aom_scale/aom_scale_rtcd.pl
index eef6f16..e32cec4 100644
--- a/aom_scale/aom_scale_rtcd.pl
+++ b/aom_scale/aom_scale_rtcd.pl
@@ -45,6 +45,8 @@
add_proto qw/void aom_yv12_partial_copy_v/, "const struct yv12_buffer_config *src_bc, int hstart1, int hend1, int vstart1, int vend1, struct yv12_buffer_config *dst_bc, int hstart2, int vstart2";
add_proto qw/void aom_yv12_partial_coloc_copy_v/, "const struct yv12_buffer_config *src_bc, struct yv12_buffer_config *dst_bc, int hstart, int hend, int vstart, int vend";
+add_proto qw/void aom_extend_frame_borders_plane_row/, "const struct yv12_buffer_config *ybf, int plane, int v_start, int v_end";
+
add_proto qw/void aom_extend_frame_borders/, "struct yv12_buffer_config *ybf, const int num_planes";
specialize qw/aom_extend_frame_borders dspr2/;
diff --git a/aom_scale/generic/yv12extend.c b/aom_scale/generic/yv12extend.c
index 5d797c8..997ff54 100644
--- a/aom_scale/generic/yv12extend.c
+++ b/aom_scale/generic/yv12extend.c
@@ -21,19 +21,20 @@
static void extend_plane(uint8_t *const src, int src_stride, int width,
int height, int extend_top, int extend_left,
- int extend_bottom, int extend_right) {
+ int extend_bottom, int extend_right, int v_start,
+ int v_end) {
assert(src != NULL);
int i;
const int linesize = extend_left + extend_right + width;
assert(linesize <= src_stride);
/* copy the left and right most columns out */
- uint8_t *src_ptr1 = src;
- uint8_t *src_ptr2 = src + width - 1;
- uint8_t *dst_ptr1 = src - extend_left;
+ uint8_t *src_ptr1 = src + v_start * src_stride;
+ uint8_t *src_ptr2 = src + v_start * src_stride + width - 1;
+ uint8_t *dst_ptr1 = src + v_start * src_stride - extend_left;
uint8_t *dst_ptr2 = src_ptr2 + 1;
- for (i = 0; i < height; ++i) {
+ for (i = v_start; i < v_end; ++i) {
memset(dst_ptr1, src_ptr1[0], extend_left);
memset(dst_ptr2, src_ptr2[0], extend_right);
src_ptr1 += src_stride;
@@ -65,19 +66,20 @@
#if CONFIG_AV1_HIGHBITDEPTH
static void extend_plane_high(uint8_t *const src8, int src_stride, int width,
int height, int extend_top, int extend_left,
- int extend_bottom, int extend_right) {
+ int extend_bottom, int extend_right, int v_start,
+ int v_end) {
int i;
const int linesize = extend_left + extend_right + width;
assert(linesize <= src_stride);
uint16_t *src = CONVERT_TO_SHORTPTR(src8);
/* copy the left and right most columns out */
- uint16_t *src_ptr1 = src;
- uint16_t *src_ptr2 = src + width - 1;
- uint16_t *dst_ptr1 = src - extend_left;
+ uint16_t *src_ptr1 = src + v_start * src_stride;
+ uint16_t *src_ptr2 = src + v_start * src_stride + width - 1;
+ uint16_t *dst_ptr1 = src + v_start * src_stride - extend_left;
uint16_t *dst_ptr2 = src_ptr2 + 1;
- for (i = 0; i < height; ++i) {
+ for (i = v_start; i < v_end; ++i) {
aom_memset16(dst_ptr1, src_ptr1[0], extend_left);
aom_memset16(dst_ptr2, src_ptr2[0], extend_right);
src_ptr1 += src_stride;
@@ -107,6 +109,41 @@
}
#endif // CONFIG_AV1_HIGHBITDEPTH
+void aom_extend_frame_borders_plane_row_c(const YV12_BUFFER_CONFIG *ybf,
+ int plane, int v_start, int v_end) {
+ const int ext_size = ybf->border;
+ const int ss_x = ybf->subsampling_x;
+ const int ss_y = ybf->subsampling_y;
+
+ assert(ybf->y_height - ybf->y_crop_height < 16);
+ assert(ybf->y_width - ybf->y_crop_width < 16);
+ assert(ybf->y_height - ybf->y_crop_height >= 0);
+ assert(ybf->y_width - ybf->y_crop_width >= 0);
+
+ const int is_uv = plane > 0;
+ const int top = ext_size >> (is_uv ? ss_y : 0);
+ const int left = ext_size >> (is_uv ? ss_x : 0);
+ const int bottom = top + ybf->heights[is_uv] - ybf->crop_heights[is_uv];
+ const int right = left + ybf->widths[is_uv] - ybf->crop_widths[is_uv];
+ const int extend_top_border = (v_start == 0);
+ const int extend_bottom_border = (v_end == ybf->crop_heights[is_uv]);
+
+#if CONFIG_AV1_HIGHBITDEPTH
+ if (ybf->flags & YV12_FLAG_HIGHBITDEPTH) {
+ extend_plane_high(ybf->buffers[plane], ybf->strides[is_uv],
+ ybf->crop_widths[is_uv], ybf->crop_heights[is_uv],
+ extend_top_border ? top : 0, left,
+ extend_bottom_border ? bottom : 0, right, v_start, v_end);
+ return;
+ }
+#endif
+
+ extend_plane(ybf->buffers[plane], ybf->strides[is_uv],
+ ybf->crop_widths[is_uv], ybf->crop_heights[is_uv],
+ extend_top_border ? top : 0, left,
+ extend_bottom_border ? bottom : 0, right, v_start, v_end);
+}
+
void aom_yv12_extend_frame_borders_c(YV12_BUFFER_CONFIG *ybf,
const int num_planes) {
assert(ybf->border % 2 == 0);
@@ -124,7 +161,8 @@
ybf->buffers[plane], ybf->strides[is_uv], ybf->crop_widths[is_uv],
ybf->crop_heights[is_uv], plane_border, plane_border,
plane_border + ybf->heights[is_uv] - ybf->crop_heights[is_uv],
- plane_border + ybf->widths[is_uv] - ybf->crop_widths[is_uv]);
+ plane_border + ybf->widths[is_uv] - ybf->crop_widths[is_uv], 0,
+ ybf->crop_heights[is_uv]);
}
return;
}
@@ -137,7 +175,8 @@
ybf->crop_widths[is_uv], ybf->crop_heights[is_uv],
plane_border, plane_border,
plane_border + ybf->heights[is_uv] - ybf->crop_heights[is_uv],
- plane_border + ybf->widths[is_uv] - ybf->crop_widths[is_uv]);
+ plane_border + ybf->widths[is_uv] - ybf->crop_widths[is_uv], 0,
+ ybf->crop_heights[is_uv]);
}
}
@@ -161,7 +200,7 @@
const int right = left + ybf->widths[is_uv] - ybf->crop_widths[is_uv];
extend_plane_high(ybf->buffers[plane], ybf->strides[is_uv],
ybf->crop_widths[is_uv], ybf->crop_heights[is_uv], top,
- left, bottom, right);
+ left, bottom, right, 0, ybf->crop_heights[is_uv]);
}
return;
}
@@ -175,7 +214,7 @@
const int right = left + ybf->widths[is_uv] - ybf->crop_widths[is_uv];
extend_plane(ybf->buffers[plane], ybf->strides[is_uv],
ybf->crop_widths[is_uv], ybf->crop_heights[is_uv], top, left,
- bottom, right);
+ bottom, right, 0, ybf->crop_heights[is_uv]);
}
}
@@ -199,17 +238,17 @@
assert(ybf->y_width - ybf->y_crop_width >= 0);
#if CONFIG_AV1_HIGHBITDEPTH
if (ybf->flags & YV12_FLAG_HIGHBITDEPTH) {
- extend_plane_high(ybf->y_buffer, ybf->y_stride, ybf->y_crop_width,
- ybf->y_crop_height, ext_size, ext_size,
- ext_size + ybf->y_height - ybf->y_crop_height,
- ext_size + ybf->y_width - ybf->y_crop_width);
+ extend_plane_high(
+ ybf->y_buffer, ybf->y_stride, ybf->y_crop_width, ybf->y_crop_height,
+ ext_size, ext_size, ext_size + ybf->y_height - ybf->y_crop_height,
+ ext_size + ybf->y_width - ybf->y_crop_width, 0, ybf->y_crop_height);
return;
}
#endif
- extend_plane(ybf->y_buffer, ybf->y_stride, ybf->y_crop_width,
- ybf->y_crop_height, ext_size, ext_size,
- ext_size + ybf->y_height - ybf->y_crop_height,
- ext_size + ybf->y_width - ybf->y_crop_width);
+ extend_plane(
+ ybf->y_buffer, ybf->y_stride, ybf->y_crop_width, ybf->y_crop_height,
+ ext_size, ext_size, ext_size + ybf->y_height - ybf->y_crop_height,
+ ext_size + ybf->y_width - ybf->y_crop_width, 0, ybf->y_crop_height);
}
#if CONFIG_AV1_HIGHBITDEPTH
diff --git a/apps/aomenc.c b/apps/aomenc.c
index 246bb69..60bc01a 100644
--- a/apps/aomenc.c
+++ b/apps/aomenc.c
@@ -451,6 +451,7 @@
&g_av1_codec_arg_defs.fwd_kf_dist,
&g_av1_codec_arg_defs.strict_level_conformance,
&g_av1_codec_arg_defs.dist_metric,
+ &g_av1_codec_arg_defs.kf_max_pyr_height,
NULL,
};
diff --git a/av1/arg_defs.c b/av1/arg_defs.c
index 58e6f4e..a954b88 100644
--- a/av1/arg_defs.c
+++ b/av1/arg_defs.c
@@ -672,5 +672,11 @@
ARG_DEF(NULL, "strict-level-conformance", 1,
"When set to 1, exit the encoder when it fails to encode "
"to a given target level"),
+ .kf_max_pyr_height = ARG_DEF(
+ NULL, "kf-max-pyr-height", 1,
+ "Maximum height of pyramid structure used for the GOP starting with a "
+ "key frame (-1 to 5). When set to -1 (default), it does not have any "
+ "effect. The actual maximum pyramid height will be the minimum of this "
+ "value and the value of gf_max_pyr_height."),
#endif // CONFIG_AV1_ENCODER
};
diff --git a/av1/arg_defs.h b/av1/arg_defs.h
index 5c30f0d..812df27 100644
--- a/av1/arg_defs.h
+++ b/av1/arg_defs.h
@@ -231,6 +231,7 @@
arg_def_t second_pass_log;
arg_def_t auto_intra_tools_off;
arg_def_t strict_level_conformance;
+ arg_def_t kf_max_pyr_height;
#endif // CONFIG_AV1_ENCODER
} av1_codec_arg_definitions_t;
diff --git a/av1/av1.cmake b/av1/av1.cmake
index c017427..d689330 100644
--- a/av1/av1.cmake
+++ b/av1/av1.cmake
@@ -364,7 +364,12 @@
"${AOM_ROOT}/av1/encoder/arm/neon/encodetxb_neon.c"
"${AOM_ROOT}/av1/encoder/arm/neon/hybrid_fwd_txfm_neon.c"
"${AOM_ROOT}/av1/encoder/arm/neon/av1_fwd_txfm2d_neon.c"
- "${AOM_ROOT}/av1/encoder/arm/neon/highbd_fwd_txfm_neon.c")
+ "${AOM_ROOT}/av1/encoder/arm/neon/highbd_fwd_txfm_neon.c"
+ "${AOM_ROOT}/av1/encoder/arm/neon/wedge_utils_neon.c"
+ "${AOM_ROOT}/av1/encoder/arm/neon/temporal_filter_neon.c")
+
+list(APPEND AOM_AV1_ENCODER_INTRIN_ARM_CRC32
+ "${AOM_ROOT}/av1/encoder/arm/crc32/hash_crc32.c")
list(APPEND AOM_AV1_ENCODER_INTRIN_MSA
"${AOM_ROOT}/av1/encoder/mips/msa/error_msa.c"
@@ -631,6 +636,16 @@
"AOM_AV1_ENCODER_INTRIN_NEON")
endif()
endif()
+
+ if(HAVE_ARM_CRC32)
+ if(CONFIG_AV1_ENCODER)
+ if(AOM_AV1_ENCODER_INTRIN_ARM_CRC32)
+ add_intrinsics_object_library("${AOM_ARM_CRC32_FLAG}" "crc32"
+ "aom_av1_encoder"
+ "AOM_AV1_ENCODER_INTRIN_ARM_CRC32")
+ endif()
+ endif()
+ endif()
endif()
if(HAVE_VSX)
diff --git a/av1/av1_cx_iface.c b/av1/av1_cx_iface.c
index 872f568..61e7ed6 100644
--- a/av1/av1_cx_iface.c
+++ b/av1/av1_cx_iface.c
@@ -188,6 +188,7 @@
// "--enable_diagonal_intra".
int auto_intra_tools_off;
int strict_level_conformance;
+ int kf_max_pyr_height;
};
#if CONFIG_REALTIME_ONLY
@@ -348,6 +349,7 @@
NULL, // second_pass_log
0, // auto_intra_tools_off
0, // strict_level_conformance
+ -1, // kf_max_pyr_height
};
#else
static const struct av1_extracfg default_extra_cfg = {
@@ -494,6 +496,7 @@
NULL, // second_pass_log
0, // auto_intra_tools_off
0, // strict_level_conformance
+ -1, // kf_max_pyr_height
};
#endif
@@ -612,8 +615,16 @@
static aom_codec_err_t validate_config(aom_codec_alg_priv_t *ctx,
const aom_codec_enc_cfg_t *cfg,
const struct av1_extracfg *extra_cfg) {
- RANGE_CHECK(cfg, g_w, 1, 65536); // 16 bits available
- RANGE_CHECK(cfg, g_h, 1, 65536); // 16 bits available
+ RANGE_CHECK(cfg, g_w, 1, 65536); // 16 bits available
+ RANGE_CHECK(cfg, g_h, 1, 65536); // 16 bits available
+ RANGE_CHECK_HI(cfg, g_forced_max_frame_width, 65536); // 16 bits available
+ RANGE_CHECK_HI(cfg, g_forced_max_frame_height, 65536); // 16 bits available
+ if (cfg->g_forced_max_frame_width) {
+ RANGE_CHECK_HI(cfg, g_w, cfg->g_forced_max_frame_width);
+ }
+ if (cfg->g_forced_max_frame_height) {
+ RANGE_CHECK_HI(cfg, g_h, cfg->g_forced_max_frame_height);
+ }
RANGE_CHECK(cfg, g_timebase.den, 1, 1000000000);
RANGE_CHECK(cfg, g_timebase.num, 1, cfg->g_timebase.den);
RANGE_CHECK_HI(cfg, g_profile, MAX_PROFILES - 1);
@@ -840,6 +851,14 @@
RANGE_CHECK_BOOL(extra_cfg, auto_intra_tools_off);
RANGE_CHECK_BOOL(extra_cfg, strict_level_conformance);
+ RANGE_CHECK(extra_cfg, kf_max_pyr_height, -1, 5);
+ if (extra_cfg->kf_max_pyr_height != -1 &&
+ extra_cfg->kf_max_pyr_height < (int)extra_cfg->gf_min_pyr_height) {
+ ERROR(
+ "The value of kf-max-pyr-height should not be smaller than "
+ "gf-min-pyr-height");
+ }
+
return AOM_CODEC_OK;
}
@@ -920,11 +939,10 @@
struct av1_extracfg *extra_cfg) {
extra_cfg->enable_cdef = (cfg->disable_cdef == 0) ? 1 : 0;
extra_cfg->enable_restoration = (cfg->disable_lr == 0);
- extra_cfg->superblock_size = (cfg->super_block_size == 64)
- ? AOM_SUPERBLOCK_SIZE_64X64
- : (cfg->super_block_size == 128)
- ? AOM_SUPERBLOCK_SIZE_128X128
- : AOM_SUPERBLOCK_SIZE_DYNAMIC;
+ extra_cfg->superblock_size =
+ (cfg->super_block_size == 64) ? AOM_SUPERBLOCK_SIZE_64X64
+ : (cfg->super_block_size == 128) ? AOM_SUPERBLOCK_SIZE_128X128
+ : AOM_SUPERBLOCK_SIZE_DYNAMIC;
extra_cfg->enable_warped_motion = (cfg->disable_warp_motion == 0);
extra_cfg->enable_dist_wtd_comp = (cfg->disable_dist_wtd_comp == 0);
extra_cfg->enable_diff_wtd_comp = (cfg->disable_diff_wtd_comp == 0);
@@ -1270,10 +1288,10 @@
tile_cfg->tile_width_count = AOMMIN(cfg->tile_width_count, MAX_TILE_COLS);
tile_cfg->tile_height_count = AOMMIN(cfg->tile_height_count, MAX_TILE_ROWS);
for (int i = 0; i < tile_cfg->tile_width_count; i++) {
- tile_cfg->tile_widths[i] = AOMMAX(cfg->tile_widths[i], 1);
+ tile_cfg->tile_widths[i] = cfg->tile_widths[i];
}
for (int i = 0; i < tile_cfg->tile_height_count; i++) {
- tile_cfg->tile_heights[i] = AOMMAX(cfg->tile_heights[i], 1);
+ tile_cfg->tile_heights[i] = cfg->tile_heights[i];
}
tile_cfg->enable_ext_tile_debug = extra_cfg->ext_tile_debug;
@@ -1418,6 +1436,8 @@
oxcf->strict_level_conformance = extra_cfg->strict_level_conformance;
+ oxcf->kf_max_pyr_height = extra_cfg->kf_max_pyr_height;
+
return AOM_CODEC_OK;
}
@@ -2698,6 +2718,25 @@
return flags;
}
+static INLINE int get_src_border_in_pixels(AV1_COMP *cpi, BLOCK_SIZE sb_size) {
+ if (cpi->oxcf.mode != REALTIME || av1_is_resize_needed(&cpi->oxcf))
+ return cpi->oxcf.border_in_pixels;
+
+ const int sb_size_in_pixels_log2 = mi_size_wide_log2[sb_size] + MI_SIZE_LOG2;
+ const int sb_aligned_width =
+ ALIGN_POWER_OF_TWO(cpi->oxcf.frm_dim_cfg.width, sb_size_in_pixels_log2);
+ const int sb_aligned_height =
+ ALIGN_POWER_OF_TWO(cpi->oxcf.frm_dim_cfg.height, sb_size_in_pixels_log2);
+ // Align the border pixels to a multiple of 32.
+ const int border_pixels_width =
+ ALIGN_POWER_OF_TWO(sb_aligned_width - cpi->oxcf.frm_dim_cfg.width, 5);
+ const int border_pixels_height =
+ ALIGN_POWER_OF_TWO(sb_aligned_height - cpi->oxcf.frm_dim_cfg.height, 5);
+ const int border_in_pixels =
+ AOMMAX(AOMMAX(border_pixels_width, border_pixels_height), 32);
+ return border_in_pixels;
+}
+
// TODO(Mufaddal): Check feasibility of abstracting functions related to LAP
// into a separate function.
static aom_codec_err_t encoder_encode(aom_codec_alg_priv_t *ctx,
@@ -2724,9 +2763,10 @@
if (img != NULL) {
res = validate_img(ctx, img);
if (res == AOM_CODEC_OK) {
- const size_t uncompressed_frame_sz = ALIGN_POWER_OF_TWO(ctx->cfg.g_w, 5) *
- ALIGN_POWER_OF_TWO(ctx->cfg.g_h, 5) *
- av1_get_image_bps(img) / 8;
+ const size_t uncompressed_frame_sz =
+ ALIGN_POWER_OF_TWO_UNSIGNED(ctx->cfg.g_w, 5) *
+ ALIGN_POWER_OF_TWO_UNSIGNED(ctx->cfg.g_h, 5) *
+ av1_get_image_bps(img) / 8;
// Due to the presence of no-show frames, the ctx->cx_data buffer holds
// compressed data corresponding to multiple frames. As no-show frames are
@@ -2816,6 +2856,10 @@
if (res == AOM_CODEC_OK) {
AV1_COMP *cpi = ppi->cpi;
+ const int num_layers =
+ cpi->svc.number_spatial_layers * cpi->svc.number_temporal_layers;
+ av1_alloc_layer_context(cpi, num_layers);
+
// Set up internal flags
if (ctx->base.init_flags & AOM_CODEC_USE_PSNR) ppi->b_calculate_psnr = 1;
@@ -2856,10 +2900,11 @@
ppi->parallel_cpi[i]->oxcf.border_in_pixels = oxcf->border_in_pixels;
}
+ const int src_border_in_pixels = get_src_border_in_pixels(cpi, sb_size);
ppi->lookahead = av1_lookahead_init(
cpi->oxcf.frm_dim_cfg.width, cpi->oxcf.frm_dim_cfg.height,
subsampling_x, subsampling_y, use_highbitdepth, lag_in_frames,
- cpi->oxcf.border_in_pixels, cpi->common.features.byte_alignment,
+ src_border_in_pixels, cpi->common.features.byte_alignment,
ctx->num_lap_buffers, (cpi->oxcf.kf_cfg.key_freq_max == 0),
cpi->oxcf.tool_cfg.enable_global_motion);
}
@@ -2879,7 +2924,7 @@
// key frame flag when we actually encode this frame.
if (av1_receive_raw_frame(cpi, flags | ctx->next_frame_flags, &sd,
src_time_stamp, src_end_time_stamp)) {
- res = update_error_state(ctx, &ppi->error);
+ res = update_error_state(ctx, cpi->common.error);
}
ctx->next_frame_flags = 0;
}
@@ -3286,7 +3331,7 @@
if (mode) {
const int res = av1_set_internal_size(
&ctx->ppi->cpi->oxcf, &ctx->ppi->cpi->resize_pending_params,
- (AOM_SCALING)mode->h_scaling_mode, (AOM_SCALING)mode->v_scaling_mode);
+ mode->h_scaling_mode, mode->v_scaling_mode);
av1_check_fpmt_config(ctx->ppi, &ctx->ppi->cpi->oxcf);
return (res == 0) ? AOM_CODEC_OK : AOM_CODEC_INVALID_PARAM;
} else {
@@ -3336,6 +3381,10 @@
if (ppi->number_spatial_layers > 1 || ppi->number_temporal_layers > 1) {
unsigned int sl, tl;
ctx->ppi->use_svc = 1;
+ const int num_layers =
+ ppi->number_spatial_layers * ppi->number_temporal_layers;
+ av1_alloc_layer_context(cpi, num_layers);
+
for (sl = 0; sl < ppi->number_spatial_layers; ++sl) {
for (tl = 0; tl < ppi->number_temporal_layers; ++tl) {
const int layer = LAYER_IDS_TO_IDX(sl, tl, ppi->number_temporal_layers);
@@ -3370,13 +3419,13 @@
AV1_COMP *const cpi = ctx->ppi->cpi;
aom_svc_ref_frame_config_t *const data =
va_arg(args, aom_svc_ref_frame_config_t *);
- cpi->svc.set_ref_frame_config = 1;
+ cpi->rtc_ref.set_ref_frame_config = 1;
for (unsigned int i = 0; i < INTER_REFS_PER_FRAME; ++i) {
- cpi->svc.reference[i] = data->reference[i];
- cpi->svc.ref_idx[i] = data->ref_idx[i];
+ cpi->rtc_ref.reference[i] = data->reference[i];
+ cpi->rtc_ref.ref_idx[i] = data->ref_idx[i];
}
for (unsigned int i = 0; i < REF_FRAMES; ++i)
- cpi->svc.refresh[i] = data->refresh[i];
+ cpi->rtc_ref.refresh[i] = data->refresh[i];
cpi->svc.use_flexible_mode = 1;
cpi->svc.ksvc_fixed_mode = 0;
return AOM_CODEC_OK;
@@ -3387,9 +3436,9 @@
AV1_COMP *const cpi = ctx->ppi->cpi;
aom_svc_ref_frame_comp_pred_t *const data =
va_arg(args, aom_svc_ref_frame_comp_pred_t *);
- cpi->svc.ref_frame_comp[0] = data->use_comp_pred[0];
- cpi->svc.ref_frame_comp[1] = data->use_comp_pred[1];
- cpi->svc.ref_frame_comp[2] = data->use_comp_pred[2];
+ cpi->rtc_ref.ref_frame_comp[0] = data->use_comp_pred[0];
+ cpi->rtc_ref.ref_frame_comp[1] = data->use_comp_pred[1];
+ cpi->rtc_ref.ref_frame_comp[2] = data->use_comp_pred[2];
return AOM_CODEC_OK;
}
@@ -3896,6 +3945,17 @@
&g_av1_codec_arg_defs.strict_level_conformance,
argv, err_string)) {
extra_cfg.strict_level_conformance = arg_parse_int_helper(&arg, err_string);
+ } else if (arg_match_helper(&arg, &g_av1_codec_arg_defs.kf_max_pyr_height,
+ argv, err_string)) {
+ extra_cfg.kf_max_pyr_height = arg_parse_int_helper(&arg, err_string);
+ } else if (arg_match_helper(&arg, &g_av1_codec_arg_defs.tile_width, argv,
+ err_string)) {
+ ctx->cfg.tile_width_count = arg_parse_list_helper(
+ &arg, ctx->cfg.tile_widths, MAX_TILE_WIDTHS, err_string);
+ } else if (arg_match_helper(&arg, &g_av1_codec_arg_defs.tile_height, argv,
+ err_string)) {
+ ctx->cfg.tile_height_count = arg_parse_list_helper(
+ &arg, ctx->cfg.tile_heights, MAX_TILE_HEIGHTS, err_string);
} else {
match = 0;
snprintf(err_string, ARG_ERR_MSG_MAX_LEN, "Cannot find aom option %s",
diff --git a/av1/av1_dx_iface.c b/av1/av1_dx_iface.c
index cb5c6e3..2567475 100644
--- a/av1/av1_dx_iface.c
+++ b/av1/av1_dx_iface.c
@@ -745,8 +745,8 @@
aom_film_grain_t *grain_params) {
if (!grain_params->apply_grain) return img;
- const int w_even = ALIGN_POWER_OF_TWO(img->d_w, 1);
- const int h_even = ALIGN_POWER_OF_TWO(img->d_h, 1);
+ const int w_even = ALIGN_POWER_OF_TWO_UNSIGNED(img->d_w, 1);
+ const int h_even = ALIGN_POWER_OF_TWO_UNSIGNED(img->d_h, 1);
BufferPool *const pool = ctx->buffer_pool;
aom_codec_frame_buffer_t *fb =
diff --git a/av1/av1_iface_common.h b/av1/av1_iface_common.h
index 57dd1b8..b923c3d 100644
--- a/av1/av1_iface_common.h
+++ b/av1/av1_iface_common.h
@@ -137,7 +137,7 @@
// Note(yunqing): if img is allocated the same as the frame buffer, y_stride
// is 32-byte aligned. Also, handle the cases while allocating img without a
// border or stride_align is less than 32.
- int border = (yv12->y_stride - (int)((img->w + 31) & ~31)) / 2;
+ int border = (yv12->y_stride - (int)((img->w + 31) & ~31u)) / 2;
yv12->border = (border < 0) ? 0 : border;
yv12->subsampling_x = img->x_chroma_shift;
yv12->subsampling_y = img->y_chroma_shift;
diff --git a/av1/common/alloccommon.c b/av1/common/alloccommon.c
index 5cf6c0f..e373dc1 100644
--- a/av1/common/alloccommon.c
+++ b/av1/common/alloccommon.c
@@ -28,8 +28,8 @@
const int mi_cols = aligned_width >> MI_SIZE_LOG2;
const int mi_rows = aligned_height >> MI_SIZE_LOG2;
- const int mb_cols = (mi_cols + 2) >> 2;
- const int mb_rows = (mi_rows + 2) >> 2;
+ const int mb_cols = ROUND_POWER_OF_TWO(mi_cols, 2);
+ const int mb_rows = ROUND_POWER_OF_TWO(mi_rows, 2);
return mb_rows * mb_cols;
}
diff --git a/av1/common/arm/av1_inv_txfm_neon.c b/av1/common/arm/av1_inv_txfm_neon.c
index bee496a..1628cbf 100644
--- a/av1/common/arm/av1_inv_txfm_neon.c
+++ b/av1/common/arm/av1_inv_txfm_neon.c
@@ -250,8 +250,7 @@
static INLINE int16x4_t set_s16x4_neon(const int16_t c0, const int16_t c1,
const int16_t c2, const int16_t c3) {
- int16x4_t val = vdup_n_s16((int16_t)0);
- val = vset_lane_s16(c0, val, 0);
+ int16x4_t val = vdup_n_s16(c0);
val = vset_lane_s16(c1, val, 1);
val = vset_lane_s16(c2, val, 2);
val = vset_lane_s16(c3, val, 3);
diff --git a/av1/common/arm/convolve_neon.c b/av1/common/arm/convolve_neon.c
index f0e4bed..2d35669 100644
--- a/av1/common/arm/convolve_neon.c
+++ b/av1/common/arm/convolve_neon.c
@@ -27,68 +27,41 @@
const int16x4_t s2, const int16x4_t s3,
const int16x4_t s4, const int16x4_t s5,
const int16x4_t s6, const int16x4_t s7,
- const int16_t *filter) {
+ const int16x8_t filter) {
+ const int16x4_t filter_lo = vget_low_s16(filter);
+ const int16x4_t filter_hi = vget_high_s16(filter);
int16x4_t sum;
- sum = vmul_n_s16(s0, filter[0]);
- sum = vmla_n_s16(sum, s1, filter[1]);
- sum = vmla_n_s16(sum, s2, filter[2]);
- sum = vmla_n_s16(sum, s5, filter[5]);
- sum = vmla_n_s16(sum, s6, filter[6]);
- sum = vmla_n_s16(sum, s7, filter[7]);
- /* filter[3] can take a max value of 128. So the max value of the result :
- * 128*255 + sum > 16 bits
- */
- sum = vqadd_s16(sum, vmul_n_s16(s3, filter[3]));
- sum = vqadd_s16(sum, vmul_n_s16(s4, filter[4]));
+ sum = vmul_lane_s16(s0, filter_lo, 0);
+ sum = vmla_lane_s16(sum, s1, filter_lo, 1);
+ sum = vmla_lane_s16(sum, s2, filter_lo, 2);
+ sum = vmla_lane_s16(sum, s3, filter_lo, 3);
+ sum = vmla_lane_s16(sum, s4, filter_hi, 0);
+ sum = vmla_lane_s16(sum, s5, filter_hi, 1);
+ sum = vmla_lane_s16(sum, s6, filter_hi, 2);
+ sum = vmla_lane_s16(sum, s7, filter_hi, 3);
return sum;
}
-static INLINE uint8x8_t convolve8_horiz_8x8(
- const int16x8_t s0, const int16x8_t s1, const int16x8_t s2,
- const int16x8_t s3, const int16x8_t s4, const int16x8_t s5,
- const int16x8_t s6, const int16x8_t s7, const int16_t *filter,
- const int16x8_t shift_round_0, const int16x8_t shift_by_bits) {
- int16x8_t sum;
-
- sum = vmulq_n_s16(s0, filter[0]);
- sum = vmlaq_n_s16(sum, s1, filter[1]);
- sum = vmlaq_n_s16(sum, s2, filter[2]);
- sum = vmlaq_n_s16(sum, s5, filter[5]);
- sum = vmlaq_n_s16(sum, s6, filter[6]);
- sum = vmlaq_n_s16(sum, s7, filter[7]);
- /* filter[3] can take a max value of 128. So the max value of the result :
- * 128*255 + sum > 16 bits
- */
- sum = vqaddq_s16(sum, vmulq_n_s16(s3, filter[3]));
- sum = vqaddq_s16(sum, vmulq_n_s16(s4, filter[4]));
-
- sum = vqrshlq_s16(sum, shift_round_0);
- sum = vqrshlq_s16(sum, shift_by_bits);
-
- return vqmovun_s16(sum);
-}
-
#if !defined(__aarch64__)
static INLINE uint8x8_t convolve8_horiz_4x1(
const int16x4_t s0, const int16x4_t s1, const int16x4_t s2,
const int16x4_t s3, const int16x4_t s4, const int16x4_t s5,
- const int16x4_t s6, const int16x4_t s7, const int16_t *filter,
+ const int16x4_t s6, const int16x4_t s7, const int16x8_t filter,
const int16x4_t shift_round_0, const int16x4_t shift_by_bits) {
+ const int16x4_t filter_lo = vget_low_s16(filter);
+ const int16x4_t filter_hi = vget_high_s16(filter);
int16x4_t sum;
- sum = vmul_n_s16(s0, filter[0]);
- sum = vmla_n_s16(sum, s1, filter[1]);
- sum = vmla_n_s16(sum, s2, filter[2]);
- sum = vmla_n_s16(sum, s5, filter[5]);
- sum = vmla_n_s16(sum, s6, filter[6]);
- sum = vmla_n_s16(sum, s7, filter[7]);
- /* filter[3] can take a max value of 128. So the max value of the result :
- * 128*255 + sum > 16 bits
- */
- sum = vqadd_s16(sum, vmul_n_s16(s3, filter[3]));
- sum = vqadd_s16(sum, vmul_n_s16(s4, filter[4]));
+ sum = vmul_lane_s16(s0, filter_lo, 0);
+ sum = vmla_lane_s16(sum, s1, filter_lo, 1);
+ sum = vmla_lane_s16(sum, s2, filter_lo, 2);
+ sum = vmla_lane_s16(sum, s3, filter_lo, 3);
+ sum = vmla_lane_s16(sum, s4, filter_hi, 0);
+ sum = vmla_lane_s16(sum, s5, filter_hi, 1);
+ sum = vmla_lane_s16(sum, s6, filter_hi, 2);
+ sum = vmla_lane_s16(sum, s7, filter_hi, 3);
sum = vqrshl_s16(sum, shift_round_0);
sum = vqrshl_s16(sum, shift_by_bits);
@@ -100,80 +73,77 @@
static INLINE uint8x8_t convolve8_vert_8x4(
const int16x8_t s0, const int16x8_t s1, const int16x8_t s2,
const int16x8_t s3, const int16x8_t s4, const int16x8_t s5,
- const int16x8_t s6, const int16x8_t s7, const int16_t *filter) {
+ const int16x8_t s6, const int16x8_t s7, const int16x8_t filter) {
+ const int16x4_t filter_lo = vget_low_s16(filter);
+ const int16x4_t filter_hi = vget_high_s16(filter);
int16x8_t sum;
- sum = vmulq_n_s16(s0, filter[0]);
- sum = vmlaq_n_s16(sum, s1, filter[1]);
- sum = vmlaq_n_s16(sum, s2, filter[2]);
- sum = vmlaq_n_s16(sum, s5, filter[5]);
- sum = vmlaq_n_s16(sum, s6, filter[6]);
- sum = vmlaq_n_s16(sum, s7, filter[7]);
- /* filter[3] can take a max value of 128. So the max value of the result :
- * 128*255 + sum > 16 bits
- */
- sum = vqaddq_s16(sum, vmulq_n_s16(s3, filter[3]));
- sum = vqaddq_s16(sum, vmulq_n_s16(s4, filter[4]));
+ sum = vmulq_lane_s16(s0, filter_lo, 0);
+ sum = vmlaq_lane_s16(sum, s1, filter_lo, 1);
+ sum = vmlaq_lane_s16(sum, s2, filter_lo, 2);
+ sum = vmlaq_lane_s16(sum, s3, filter_lo, 3);
+ sum = vmlaq_lane_s16(sum, s4, filter_hi, 0);
+ sum = vmlaq_lane_s16(sum, s5, filter_hi, 1);
+ sum = vmlaq_lane_s16(sum, s6, filter_hi, 2);
+ sum = vmlaq_lane_s16(sum, s7, filter_hi, 3);
- return vqrshrun_n_s16(sum, FILTER_BITS);
+ return vqrshrun_n_s16(sum, FILTER_BITS - 1);
}
-static INLINE uint16x4_t convolve8_vert_4x4_s32(
+static INLINE int16x4_t convolve8_vert_4x4_s32(
const int16x4_t s0, const int16x4_t s1, const int16x4_t s2,
const int16x4_t s3, const int16x4_t s4, const int16x4_t s5,
- const int16x4_t s6, const int16x4_t s7, const int16_t *y_filter,
+ const int16x4_t s6, const int16x4_t s7, const int16x8_t y_filter,
const int32x4_t round_shift_vec, const int32x4_t offset_const,
const int32x4_t sub_const_vec) {
- int32x4_t sum0;
- uint16x4_t res;
- const int32x4_t zero = vdupq_n_s32(0);
+ const int16x4_t y_filter_lo = vget_low_s16(y_filter);
+ const int16x4_t y_filter_hi = vget_high_s16(y_filter);
+ int32x4_t sum;
- sum0 = vmull_n_s16(s0, y_filter[0]);
- sum0 = vmlal_n_s16(sum0, s1, y_filter[1]);
- sum0 = vmlal_n_s16(sum0, s2, y_filter[2]);
- sum0 = vmlal_n_s16(sum0, s3, y_filter[3]);
- sum0 = vmlal_n_s16(sum0, s4, y_filter[4]);
- sum0 = vmlal_n_s16(sum0, s5, y_filter[5]);
- sum0 = vmlal_n_s16(sum0, s6, y_filter[6]);
- sum0 = vmlal_n_s16(sum0, s7, y_filter[7]);
+ sum = vmull_lane_s16(s0, y_filter_lo, 0);
+ sum = vmlal_lane_s16(sum, s1, y_filter_lo, 1);
+ sum = vmlal_lane_s16(sum, s2, y_filter_lo, 2);
+ sum = vmlal_lane_s16(sum, s3, y_filter_lo, 3);
+ sum = vmlal_lane_s16(sum, s4, y_filter_hi, 0);
+ sum = vmlal_lane_s16(sum, s5, y_filter_hi, 1);
+ sum = vmlal_lane_s16(sum, s6, y_filter_hi, 2);
+ sum = vmlal_lane_s16(sum, s7, y_filter_hi, 3);
- sum0 = vaddq_s32(sum0, offset_const);
- sum0 = vqrshlq_s32(sum0, round_shift_vec);
- sum0 = vsubq_s32(sum0, sub_const_vec);
- sum0 = vmaxq_s32(sum0, zero);
+ sum = vaddq_s32(sum, offset_const);
+ sum = vqrshlq_s32(sum, round_shift_vec);
+ sum = vsubq_s32(sum, sub_const_vec);
- res = vmovn_u32(vreinterpretq_u32_s32(sum0));
-
- return res;
+ return vmovn_s32(sum);
}
static INLINE uint8x8_t convolve8_vert_8x4_s32(
const int16x8_t s0, const int16x8_t s1, const int16x8_t s2,
const int16x8_t s3, const int16x8_t s4, const int16x8_t s5,
- const int16x8_t s6, const int16x8_t s7, const int16_t *y_filter,
+ const int16x8_t s6, const int16x8_t s7, const int16x8_t y_filter,
const int32x4_t round_shift_vec, const int32x4_t offset_const,
const int32x4_t sub_const_vec, const int16x8_t vec_round_bits) {
+ const int16x4_t y_filter_lo = vget_low_s16(y_filter);
+ const int16x4_t y_filter_hi = vget_high_s16(y_filter);
int32x4_t sum0, sum1;
- uint16x8_t res;
- const int32x4_t zero = vdupq_n_s32(0);
+ int16x8_t res;
- sum0 = vmull_n_s16(vget_low_s16(s0), y_filter[0]);
- sum0 = vmlal_n_s16(sum0, vget_low_s16(s1), y_filter[1]);
- sum0 = vmlal_n_s16(sum0, vget_low_s16(s2), y_filter[2]);
- sum0 = vmlal_n_s16(sum0, vget_low_s16(s3), y_filter[3]);
- sum0 = vmlal_n_s16(sum0, vget_low_s16(s4), y_filter[4]);
- sum0 = vmlal_n_s16(sum0, vget_low_s16(s5), y_filter[5]);
- sum0 = vmlal_n_s16(sum0, vget_low_s16(s6), y_filter[6]);
- sum0 = vmlal_n_s16(sum0, vget_low_s16(s7), y_filter[7]);
+ sum0 = vmull_lane_s16(vget_low_s16(s0), y_filter_lo, 0);
+ sum0 = vmlal_lane_s16(sum0, vget_low_s16(s1), y_filter_lo, 1);
+ sum0 = vmlal_lane_s16(sum0, vget_low_s16(s2), y_filter_lo, 2);
+ sum0 = vmlal_lane_s16(sum0, vget_low_s16(s3), y_filter_lo, 3);
+ sum0 = vmlal_lane_s16(sum0, vget_low_s16(s4), y_filter_hi, 0);
+ sum0 = vmlal_lane_s16(sum0, vget_low_s16(s5), y_filter_hi, 1);
+ sum0 = vmlal_lane_s16(sum0, vget_low_s16(s6), y_filter_hi, 2);
+ sum0 = vmlal_lane_s16(sum0, vget_low_s16(s7), y_filter_hi, 3);
- sum1 = vmull_n_s16(vget_high_s16(s0), y_filter[0]);
- sum1 = vmlal_n_s16(sum1, vget_high_s16(s1), y_filter[1]);
- sum1 = vmlal_n_s16(sum1, vget_high_s16(s2), y_filter[2]);
- sum1 = vmlal_n_s16(sum1, vget_high_s16(s3), y_filter[3]);
- sum1 = vmlal_n_s16(sum1, vget_high_s16(s4), y_filter[4]);
- sum1 = vmlal_n_s16(sum1, vget_high_s16(s5), y_filter[5]);
- sum1 = vmlal_n_s16(sum1, vget_high_s16(s6), y_filter[6]);
- sum1 = vmlal_n_s16(sum1, vget_high_s16(s7), y_filter[7]);
+ sum1 = vmull_lane_s16(vget_high_s16(s0), y_filter_lo, 0);
+ sum1 = vmlal_lane_s16(sum1, vget_high_s16(s1), y_filter_lo, 1);
+ sum1 = vmlal_lane_s16(sum1, vget_high_s16(s2), y_filter_lo, 2);
+ sum1 = vmlal_lane_s16(sum1, vget_high_s16(s3), y_filter_lo, 3);
+ sum1 = vmlal_lane_s16(sum1, vget_high_s16(s4), y_filter_hi, 0);
+ sum1 = vmlal_lane_s16(sum1, vget_high_s16(s5), y_filter_hi, 1);
+ sum1 = vmlal_lane_s16(sum1, vget_high_s16(s6), y_filter_hi, 2);
+ sum1 = vmlal_lane_s16(sum1, vget_high_s16(s7), y_filter_hi, 3);
sum0 = vaddq_s32(sum0, offset_const);
sum1 = vaddq_s32(sum1, offset_const);
@@ -181,14 +151,233 @@
sum1 = vqrshlq_s32(sum1, round_shift_vec);
sum0 = vsubq_s32(sum0, sub_const_vec);
sum1 = vsubq_s32(sum1, sub_const_vec);
- sum0 = vmaxq_s32(sum0, zero);
- sum1 = vmaxq_s32(sum1, zero);
- res = vcombine_u16(vqmovn_u32(vreinterpretq_u32_s32(sum0)),
- vqmovn_u32(vreinterpretq_u32_s32(sum1)));
- res = vqrshlq_u16(res, vec_round_bits);
+ res = vcombine_s16(vmovn_s32(sum0), vmovn_s32(sum1));
+ res = vqrshlq_s16(res, vec_round_bits);
- return vqmovn_u16(res);
+ return vqmovun_s16(res);
+}
+
+#if defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+
+static INLINE int32x4_t convolve8_4_dot(uint8x16_t samples,
+ const int8x8_t filters,
+ const int32x4_t correction,
+ const uint8x16_t range_limit,
+ const uint8x16x2_t permute_tbl) {
+ int8x16_t clamped_samples, permuted_samples[2];
+ int32x4_t sum;
+
+ /* Clamp sample range to [-128, 127] for 8-bit signed dot product. */
+ clamped_samples = vreinterpretq_s8_u8(vsubq_u8(samples, range_limit));
+
+ /* Permute samples ready for dot product. */
+ /* { 0, 1, 2, 3, 1, 2, 3, 4, 2, 3, 4, 5, 3, 4, 5, 6 } */
+ permuted_samples[0] = vqtbl1q_s8(clamped_samples, permute_tbl.val[0]);
+ /* { 4, 5, 6, 7, 5, 6, 7, 8, 6, 7, 8, 9, 7, 8, 9, 10 } */
+ permuted_samples[1] = vqtbl1q_s8(clamped_samples, permute_tbl.val[1]);
+
+ /* Accumulate dot product into 'correction' to account for range clamp. */
+ sum = vdotq_lane_s32(correction, permuted_samples[0], filters, 0);
+ sum = vdotq_lane_s32(sum, permuted_samples[1], filters, 1);
+
+ /* Narrowing and packing is performed by the caller. */
+ return sum;
+}
+
+static INLINE uint8x8_t convolve8_8_dot(
+ uint8x16_t samples, const int8x8_t filters, const int32x4_t correction,
+ const uint8x16_t range_limit, const uint8x16x3_t permute_tbl,
+ const int16x8_t shift_round_0, const int16x8_t shift_by_bits) {
+ int8x16_t clamped_samples, permuted_samples[3];
+ int32x4_t sum0, sum1;
+ int16x8_t sum;
+
+ /* Clamp sample range to [-128, 127] for 8-bit signed dot product. */
+ clamped_samples = vreinterpretq_s8_u8(vsubq_u8(samples, range_limit));
+
+ /* Permute samples ready for dot product. */
+ /* { 0, 1, 2, 3, 1, 2, 3, 4, 2, 3, 4, 5, 3, 4, 5, 6 } */
+ permuted_samples[0] = vqtbl1q_s8(clamped_samples, permute_tbl.val[0]);
+ /* { 4, 5, 6, 7, 5, 6, 7, 8, 6, 7, 8, 9, 7, 8, 9, 10 } */
+ permuted_samples[1] = vqtbl1q_s8(clamped_samples, permute_tbl.val[1]);
+ /* { 8, 9, 10, 11, 9, 10, 11, 12, 10, 11, 12, 13, 11, 12, 13, 14 } */
+ permuted_samples[2] = vqtbl1q_s8(clamped_samples, permute_tbl.val[2]);
+
+ /* Accumulate dot product into 'correction' to account for range clamp. */
+ /* First 4 output values. */
+ sum0 = vdotq_lane_s32(correction, permuted_samples[0], filters, 0);
+ sum0 = vdotq_lane_s32(sum0, permuted_samples[1], filters, 1);
+ /* Second 4 output values. */
+ sum1 = vdotq_lane_s32(correction, permuted_samples[1], filters, 0);
+ sum1 = vdotq_lane_s32(sum1, permuted_samples[2], filters, 1);
+
+ /* Narrow and re-pack. */
+ sum = vcombine_s16(vmovn_s32(sum0), vmovn_s32(sum1));
+ sum = vqrshlq_s16(sum, shift_round_0);
+ sum = vqrshlq_s16(sum, shift_by_bits);
+ return vqmovun_s16(sum);
+}
+
+void av1_convolve_x_sr_neon(const uint8_t *src, int src_stride, uint8_t *dst,
+ int dst_stride, int w, int h,
+ const InterpFilterParams *filter_params_x,
+ const int subpel_x_qn,
+ ConvolveParams *conv_params) {
+ if (filter_params_x->taps > 8) {
+ av1_convolve_x_sr_c(src, src_stride, dst, dst_stride, w, h, filter_params_x,
+ subpel_x_qn, conv_params);
+ return;
+ }
+ const uint8_t horiz_offset = filter_params_x->taps / 2 - 1;
+ const int8_t bits = FILTER_BITS - conv_params->round_0;
+
+ assert(bits >= 0);
+ assert((FILTER_BITS - conv_params->round_1) >= 0 ||
+ ((conv_params->round_0 + conv_params->round_1) == 2 * FILTER_BITS));
+
+ const int16_t *x_filter_ptr = av1_get_interp_filter_subpel_kernel(
+ filter_params_x, subpel_x_qn & SUBPEL_MASK);
+ // Filter values are even, so downshift by 1 to reduce intermediate precision
+ // requirements.
+ const int8x8_t x_filter = vshrn_n_s16(vld1q_s16(x_filter_ptr), 1);
+ // Dot product constants.
+ const int16x8_t correct_tmp = vshll_n_s8(x_filter, 7);
+ const int32x4_t correction = vdupq_n_s32(vaddlvq_s16(correct_tmp));
+ const uint8x16_t range_limit = vdupq_n_u8(128);
+
+ const int16x8_t shift_round_0 = vdupq_n_s16(-conv_params->round_0 + 1);
+ const int16x8_t shift_by_bits = vdupq_n_s16(-bits);
+
+ src -= horiz_offset;
+
+ if (w <= 4) {
+ const uint8x16x2_t permute_tbl = vld1q_u8_x2(dot_prod_permute_tbl);
+ uint8x16_t s0, s1, s2, s3;
+ int32x4_t t0, t1, t2, t3;
+ int16x8_t t01, t23;
+ uint8x8_t d01, d23;
+
+ do {
+ s0 = vld1q_u8(src + 0 * src_stride);
+ s1 = vld1q_u8(src + 1 * src_stride);
+ s2 = vld1q_u8(src + 2 * src_stride);
+ s3 = vld1q_u8(src + 3 * src_stride);
+
+ t0 = convolve8_4_dot(s0, x_filter, correction, range_limit, permute_tbl);
+ t1 = convolve8_4_dot(s1, x_filter, correction, range_limit, permute_tbl);
+ t2 = convolve8_4_dot(s2, x_filter, correction, range_limit, permute_tbl);
+ t3 = convolve8_4_dot(s3, x_filter, correction, range_limit, permute_tbl);
+
+ t01 = vcombine_s16(vmovn_s32(t0), vmovn_s32(t1));
+ t23 = vcombine_s16(vmovn_s32(t2), vmovn_s32(t3));
+
+ t01 = vqrshlq_s16(t01, shift_round_0);
+ t23 = vqrshlq_s16(t23, shift_round_0);
+
+ t01 = vqrshlq_s16(t01, shift_by_bits);
+ t23 = vqrshlq_s16(t23, shift_by_bits);
+
+ d01 = vqmovun_s16(t01);
+ d23 = vqmovun_s16(t23);
+
+ if (w == 2) {
+ vst1_lane_u16((uint16_t *)(dst + 0 * dst_stride),
+ vreinterpret_u16_u8(d01), 0);
+ vst1_lane_u16((uint16_t *)(dst + 1 * dst_stride),
+ vreinterpret_u16_u8(d01), 2);
+ if (h != 2) {
+ vst1_lane_u16((uint16_t *)(dst + 2 * dst_stride),
+ vreinterpret_u16_u8(d23), 0);
+ vst1_lane_u16((uint16_t *)(dst + 3 * dst_stride),
+ vreinterpret_u16_u8(d23), 2);
+ }
+ } else {
+ vst1_lane_u32((uint32_t *)(dst + 0 * dst_stride),
+ vreinterpret_u32_u8(d01), 0);
+ vst1_lane_u32((uint32_t *)(dst + 1 * dst_stride),
+ vreinterpret_u32_u8(d01), 1);
+ if (h != 2) {
+ vst1_lane_u32((uint32_t *)(dst + 2 * dst_stride),
+ vreinterpret_u32_u8(d23), 0);
+ vst1_lane_u32((uint32_t *)(dst + 3 * dst_stride),
+ vreinterpret_u32_u8(d23), 1);
+ }
+ }
+
+ h -= 4;
+ src += 4 * src_stride;
+ dst += 4 * dst_stride;
+ } while (h > 0);
+
+ } else {
+ const uint8x16x3_t permute_tbl = vld1q_u8_x3(dot_prod_permute_tbl);
+ uint8x16_t s0, s1, s2, s3;
+ uint8x8_t d0, d1, d2, d3;
+
+ do {
+ int width = w;
+ const uint8_t *s = src;
+ uint8_t *d = dst;
+
+ do {
+ s0 = vld1q_u8(s + 0 * src_stride);
+ s1 = vld1q_u8(s + 1 * src_stride);
+ s2 = vld1q_u8(s + 2 * src_stride);
+ s3 = vld1q_u8(s + 3 * src_stride);
+
+ d0 = convolve8_8_dot(s0, x_filter, correction, range_limit, permute_tbl,
+ shift_round_0, shift_by_bits);
+ d1 = convolve8_8_dot(s1, x_filter, correction, range_limit, permute_tbl,
+ shift_round_0, shift_by_bits);
+ d2 = convolve8_8_dot(s2, x_filter, correction, range_limit, permute_tbl,
+ shift_round_0, shift_by_bits);
+ d3 = convolve8_8_dot(s3, x_filter, correction, range_limit, permute_tbl,
+ shift_round_0, shift_by_bits);
+
+ vst1_u8(d + 0 * dst_stride, d0);
+ vst1_u8(d + 1 * dst_stride, d1);
+ if (h != 2) {
+ vst1_u8(d + 2 * dst_stride, d2);
+ vst1_u8(d + 3 * dst_stride, d3);
+ }
+
+ s += 8;
+ d += 8;
+ width -= 8;
+ } while (width > 0);
+
+ src += 4 * src_stride;
+ dst += 4 * dst_stride;
+ h -= 4;
+ } while (h > 0);
+ }
+}
+
+#else // !(defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD))
+
+static INLINE uint8x8_t convolve8_horiz_8x8(
+ const int16x8_t s0, const int16x8_t s1, const int16x8_t s2,
+ const int16x8_t s3, const int16x8_t s4, const int16x8_t s5,
+ const int16x8_t s6, const int16x8_t s7, const int16x8_t filter,
+ const int16x8_t shift_round_0, const int16x8_t shift_by_bits) {
+ const int16x4_t filter_lo = vget_low_s16(filter);
+ const int16x4_t filter_hi = vget_high_s16(filter);
+ int16x8_t sum;
+
+ sum = vmulq_lane_s16(s0, filter_lo, 0);
+ sum = vmlaq_lane_s16(sum, s1, filter_lo, 1);
+ sum = vmlaq_lane_s16(sum, s2, filter_lo, 2);
+ sum = vmlaq_lane_s16(sum, s3, filter_lo, 3);
+ sum = vmlaq_lane_s16(sum, s4, filter_hi, 0);
+ sum = vmlaq_lane_s16(sum, s5, filter_hi, 1);
+ sum = vmlaq_lane_s16(sum, s6, filter_hi, 2);
+ sum = vmlaq_lane_s16(sum, s7, filter_hi, 3);
+
+ sum = vqrshlq_s16(sum, shift_round_0);
+ sum = vqrshlq_s16(sum, shift_by_bits);
+
+ return vqmovun_s16(sum);
}
void av1_convolve_x_sr_neon(const uint8_t *src, int src_stride, uint8_t *dst,
@@ -213,10 +402,12 @@
assert((FILTER_BITS - conv_params->round_1) >= 0 ||
((conv_params->round_0 + conv_params->round_1) == 2 * FILTER_BITS));
- const int16_t *x_filter = av1_get_interp_filter_subpel_kernel(
+ const int16_t *x_filter_ptr = av1_get_interp_filter_subpel_kernel(
filter_params_x, subpel_x_qn & SUBPEL_MASK);
+ // Filter values are even so downshift by 1 to reduce precision requirements.
+ const int16x8_t x_filter = vshrq_n_s16(vld1q_s16(x_filter_ptr), 1);
- const int16x8_t shift_round_0 = vdupq_n_s16(-conv_params->round_0);
+ const int16x8_t shift_round_0 = vdupq_n_s16(-conv_params->round_0 + 1);
const int16x8_t shift_by_bits = vdupq_n_s16(-bits);
src -= horiz_offset;
@@ -609,6 +800,8 @@
#endif
}
+#endif // defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+
void av1_convolve_y_sr_neon(const uint8_t *src, int src_stride, uint8_t *dst,
int dst_stride, int w, int h,
const InterpFilterParams *filter_params_y,
@@ -622,8 +815,10 @@
src -= vert_offset * src_stride;
- const int16_t *y_filter = av1_get_interp_filter_subpel_kernel(
+ const int16_t *y_filter_ptr = av1_get_interp_filter_subpel_kernel(
filter_params_y, subpel_y_qn & SUBPEL_MASK);
+ // Filter values are even so downshift by 1 to reduce precision requirements.
+ const int16x8_t y_filter = vshrq_n_s16(vld1q_s16(y_filter_ptr), 1);
if (w <= 4) {
uint8x8_t d01;
@@ -671,8 +866,8 @@
d2 = convolve8_4x4(s2, s3, s4, s5, s6, s7, s8, s9, y_filter);
d3 = convolve8_4x4(s3, s4, s5, s6, s7, s8, s9, s10, y_filter);
- d01 = vqrshrun_n_s16(vcombine_s16(d0, d1), FILTER_BITS);
- d23 = vqrshrun_n_s16(vcombine_s16(d2, d3), FILTER_BITS);
+ d01 = vqrshrun_n_s16(vcombine_s16(d0, d1), FILTER_BITS - 1);
+ d23 = vqrshrun_n_s16(vcombine_s16(d2, d3), FILTER_BITS - 1);
if ((w == 4) && (h != 2)) {
vst1_lane_u32((uint32_t *)dst, vreinterpret_u32_u8(d01),
0); // 00 01 02 03
@@ -722,7 +917,7 @@
d0 = convolve8_4x4(s0, s1, s2, s3, s4, s5, s6, s7, y_filter);
- d01 = vqrshrun_n_s16(vcombine_s16(d0, d0), FILTER_BITS);
+ d01 = vqrshrun_n_s16(vcombine_s16(d0, d0), FILTER_BITS - 1);
if (w == 4) {
vst1_lane_u32((uint32_t *)dst, vreinterpret_u32_u8(d01), 0);
@@ -849,11 +1044,175 @@
}
}
+#if defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+
+static INLINE void av1_convolve_2d_sr_horiz_neon(
+ const uint8_t *src, int src_stride, int16_t *im_block, int im_stride, int w,
+ int im_h, const int16x8_t x_filter_s16, const int round_0) {
+ const int bd = 8;
+
+ const uint8_t *src_ptr = src;
+ int16_t *dst_ptr = im_block;
+ int dst_stride = im_stride;
+
+ int height = im_h;
+
+ // Filter values are even, so downshift by 1 to reduce intermediate precision
+ // requirements.
+ const int8x8_t x_filter = vshrn_n_s16(x_filter_s16, 1);
+ const int32_t horiz_const = (1 << (bd + FILTER_BITS - 2));
+ // Dot product constants.
+ const int16x8_t correct_tmp = vshlq_n_s16(x_filter_s16, 6);
+ const int32x4_t correction =
+ vdupq_n_s32(vaddlvq_s16(correct_tmp) + horiz_const);
+ const uint8x16_t range_limit = vdupq_n_u8(128);
+
+ assert(round_0 > 0);
+
+ if (w <= 4) {
+ const uint8x16x2_t permute_tbl = vld1q_u8_x2(dot_prod_permute_tbl);
+ const int16x4_t shift_round_0 = vdup_n_s16(-(round_0 - 1));
+ uint8x16_t s0, s1, s2, s3;
+ int32x4_t t0, t1, t2, t3;
+ int16x4_t d0, d1, d2, d3;
+
+ do {
+ assert(height >= 4);
+
+ load_u8_8x16(src_ptr, src_stride, &s0, &s1, &s2, &s3);
+
+ t0 = convolve8_4_dot_s16(s0, x_filter, correction, range_limit,
+ permute_tbl);
+ t1 = convolve8_4_dot_s16(s1, x_filter, correction, range_limit,
+ permute_tbl);
+ t2 = convolve8_4_dot_s16(s2, x_filter, correction, range_limit,
+ permute_tbl);
+ t3 = convolve8_4_dot_s16(s3, x_filter, correction, range_limit,
+ permute_tbl);
+
+ d0 = vqrshl_s16(vmovn_s32(t0), shift_round_0);
+ d1 = vqrshl_s16(vmovn_s32(t1), shift_round_0);
+ d2 = vqrshl_s16(vmovn_s32(t2), shift_round_0);
+ d3 = vqrshl_s16(vmovn_s32(t3), shift_round_0);
+
+ if (w == 2) {
+ vst1_lane_u32((uint32_t *)(dst_ptr + 0 * dst_stride),
+ vreinterpret_u32_s16(d0), 0);
+ vst1_lane_u32((uint32_t *)(dst_ptr + 1 * dst_stride),
+ vreinterpret_u32_s16(d1), 0);
+ vst1_lane_u32((uint32_t *)(dst_ptr + 2 * dst_stride),
+ vreinterpret_u32_s16(d2), 0);
+ vst1_lane_u32((uint32_t *)(dst_ptr + 3 * dst_stride),
+ vreinterpret_u32_s16(d3), 0);
+ } else {
+ vst1_s16(dst_ptr + 0 * dst_stride, d0);
+ vst1_s16(dst_ptr + 1 * dst_stride, d1);
+ vst1_s16(dst_ptr + 2 * dst_stride, d2);
+ vst1_s16(dst_ptr + 3 * dst_stride, d3);
+ }
+
+ src_ptr += 4 * src_stride;
+ dst_ptr += 4 * dst_stride;
+ height -= 4;
+ } while (height >= 4);
+
+ if (height) {
+ assert(height < 4);
+
+ do {
+ s0 = vld1q_u8(src_ptr);
+ t0 = convolve8_4_dot_s16(s0, x_filter, correction, range_limit,
+ permute_tbl);
+ d0 = vqrshl_s16(vmovn_s32(t0), shift_round_0);
+
+ if (w == 2) {
+ vst1_lane_u32((uint32_t *)dst_ptr, vreinterpret_u32_s16(d0), 0);
+ } else {
+ vst1_s16(dst_ptr, d0);
+ }
+
+ src_ptr += src_stride;
+ dst_ptr += dst_stride;
+ height--;
+ } while (height > 0);
+ }
+ } else {
+ const uint8x16x3_t permute_tbl = vld1q_u8_x3(dot_prod_permute_tbl);
+ const int16x8_t shift_round_0 = vdupq_n_s16(-(round_0 - 1));
+ uint8x16_t s0, s1, s2, s3;
+ int16x8_t d0, d1, d2, d3;
+
+ do {
+ assert(height >= 4);
+
+ const uint8_t *s = src_ptr;
+ int16_t *d = dst_ptr;
+ int width = w;
+
+ do {
+ s0 = vld1q_u8(s + 0 * src_stride);
+ s1 = vld1q_u8(s + 1 * src_stride);
+ s2 = vld1q_u8(s + 2 * src_stride);
+ s3 = vld1q_u8(s + 3 * src_stride);
+
+ d0 = convolve8_8_dot_s16(s0, x_filter, correction, range_limit,
+ permute_tbl, shift_round_0);
+ d1 = convolve8_8_dot_s16(s1, x_filter, correction, range_limit,
+ permute_tbl, shift_round_0);
+ d2 = convolve8_8_dot_s16(s2, x_filter, correction, range_limit,
+ permute_tbl, shift_round_0);
+ d3 = convolve8_8_dot_s16(s3, x_filter, correction, range_limit,
+ permute_tbl, shift_round_0);
+
+ vst1q_s16(d + 0 * dst_stride, d0);
+ vst1q_s16(d + 1 * dst_stride, d1);
+ vst1q_s16(d + 2 * dst_stride, d2);
+ vst1q_s16(d + 3 * dst_stride, d3);
+
+ s += 8;
+ d += 8;
+ width -= 8;
+ } while (width > 0);
+
+ src_ptr += 4 * src_stride;
+ dst_ptr += 4 * dst_stride;
+ height -= 4;
+ } while (height >= 4);
+
+ if (height) {
+ assert(height < 4);
+
+ do {
+ const uint8_t *s = src_ptr;
+ int16_t *d = dst_ptr;
+ int width = w;
+
+ do {
+ s0 = vld1q_u8(s);
+ d0 = convolve8_8_dot_s16(s0, x_filter, correction, range_limit,
+ permute_tbl, shift_round_0);
+ vst1q_s16(d, d0);
+
+ s += 8;
+ d += 8;
+ width -= 8;
+ } while (width > 0);
+
+ src_ptr += src_stride;
+ dst_ptr += dst_stride;
+ height--;
+ } while (height > 0);
+ }
+ }
+}
+
+#else // !(defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD))
+
// Horizontal filtering for convolve_2d_sr for width multiple of 8
// Processes one row at a time
static INLINE void horiz_filter_w8_single_row(
const uint8_t *src_ptr, int src_stride, int16_t *dst_ptr,
- const int dst_stride, int width, int height, const int16_t *x_filter,
+ const int dst_stride, int width, int height, const int16x8_t x_filter,
const int16x8_t horiz_const, const int16x8_t shift_round_0) {
int16x8_t s0, s1, s2, s3, s4, s5, s6, s7;
do {
@@ -899,7 +1258,7 @@
// Processes one row at a time
static INLINE void horiz_filter_w4_single_row(
const uint8_t *src_ptr, int src_stride, int16_t *dst_ptr,
- const int dst_stride, int width, int height, const int16_t *x_filter,
+ const int dst_stride, int width, int height, const int16x8_t x_filter,
const int16x4_t horiz_const, const int16x4_t shift_round_0) {
int16x4_t s0, s1, s2, s3, s4, s5, s6, s7;
do {
@@ -928,87 +1287,46 @@
int16x4_t d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter,
horiz_const, shift_round_0);
- if (width == 4) {
- vst1_s16(dst_ptr, d0);
- dst_ptr += dst_stride;
- } else if (width == 2) {
+ if (width == 2) {
vst1_lane_u32((uint32_t *)dst_ptr, vreinterpret_u32_s16(d0), 0);
- dst_ptr += dst_stride;
+ } else {
+ vst1_s16(dst_ptr, d0);
}
+ dst_ptr += dst_stride;
src_ptr += src_stride;
height--;
} while (height > 0);
}
-void av1_convolve_2d_sr_neon(const uint8_t *src, int src_stride, uint8_t *dst,
- int dst_stride, int w, int h,
- const InterpFilterParams *filter_params_x,
- const InterpFilterParams *filter_params_y,
- const int subpel_x_qn, const int subpel_y_qn,
- ConvolveParams *conv_params) {
- if (filter_params_x->taps > 8) {
- av1_convolve_2d_sr_c(src, src_stride, dst, dst_stride, w, h,
- filter_params_x, filter_params_y, subpel_x_qn,
- subpel_y_qn, conv_params);
- return;
- }
- int im_dst_stride;
- int width, height;
-#if defined(__aarch64__)
- uint8x8_t t0;
- uint8x8_t t1, t2, t3, t4, t5, t6, t7;
- const uint8_t *s;
-#endif
-
- DECLARE_ALIGNED(16, int16_t,
- im_block[(MAX_SB_SIZE + HORIZ_EXTRA_ROWS) * MAX_SB_SIZE]);
-
+static INLINE void av1_convolve_2d_sr_horiz_neon(
+ const uint8_t *src, int src_stride, int16_t *im_block, int im_stride, int w,
+ int im_h, const int16x8_t x_filter_s16, const int round_0) {
const int bd = 8;
- const int im_h = h + filter_params_y->taps - 1;
- const int im_stride = MAX_SB_SIZE;
- const int vert_offset = filter_params_y->taps / 2 - 1;
- const int horiz_offset = filter_params_x->taps / 2 - 1;
- const uint8_t *src_ptr = src - vert_offset * src_stride - horiz_offset;
+ const uint8_t *src_ptr = src;
+ int16_t *dst_ptr = im_block;
+ int dst_stride = im_stride;
- int16_t *dst_ptr;
+ int height = im_h;
- dst_ptr = im_block;
- im_dst_stride = im_stride;
- height = im_h;
- width = w;
+ // Filter values are even, so downshift by 1 to reduce intermediate precision
+ // requirements.
+ const int16x8_t x_filter = vshrq_n_s16(x_filter_s16, 1);
- const int16_t round_bits =
- FILTER_BITS * 2 - conv_params->round_0 - conv_params->round_1;
- const int16x8_t vec_round_bits = vdupq_n_s16(-round_bits);
- const int offset_bits = bd + 2 * FILTER_BITS - conv_params->round_0;
- const int16_t *x_filter = av1_get_interp_filter_subpel_kernel(
- filter_params_x, subpel_x_qn & SUBPEL_MASK);
-
- int16_t x_filter_tmp[8];
- int16x8_t filter_x_coef = vld1q_s16(x_filter);
-
- // filter coeffs are even, so downshifting by 1 to reduce intermediate
- // precision requirements.
- filter_x_coef = vshrq_n_s16(filter_x_coef, 1);
- vst1q_s16(&x_filter_tmp[0], filter_x_coef);
-
- assert(conv_params->round_0 > 0);
+ assert(round_0 > 0);
if (w <= 4) {
const int16x4_t horiz_const = vdup_n_s16((1 << (bd + FILTER_BITS - 2)));
- const int16x4_t shift_round_0 = vdup_n_s16(-(conv_params->round_0 - 1));
+ const int16x4_t shift_round_0 = vdup_n_s16(-(round_0 - 1));
#if defined(__aarch64__)
- int16x4_t s0, s1, s2, s3, s4, s5, s6, s7, s8, s9, s10, d0, d1, d2, d3;
do {
+ int16x4_t s0, s1, s2, s3, s4, s5, s6, s7, s8, s9, s10, d0, d1, d2, d3;
+ uint8x8_t t0, t1, t2, t3;
+ const uint8_t *s = src_ptr;
+
assert(height >= 4);
- s = src_ptr;
- __builtin_prefetch(s + 0 * src_stride);
- __builtin_prefetch(s + 1 * src_stride);
- __builtin_prefetch(s + 2 * src_stride);
- __builtin_prefetch(s + 3 * src_stride);
load_u8_8x4(s, src_stride, &t0, &t1, &t2, &t3);
transpose_u8_8x4(&t0, &t1, &t2, &t3);
@@ -1021,10 +1339,6 @@
s5 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t1)));
s6 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t2)));
- __builtin_prefetch(dst_ptr + 0 * im_dst_stride);
- __builtin_prefetch(dst_ptr + 1 * im_dst_stride);
- __builtin_prefetch(dst_ptr + 2 * im_dst_stride);
- __builtin_prefetch(dst_ptr + 3 * im_dst_stride);
s += 7;
load_u8_8x4(s, src_stride, &t0, &t1, &t2, &t3);
@@ -1035,68 +1349,65 @@
s9 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t2)));
s10 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t3)));
- d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter_tmp,
+ d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter,
horiz_const, shift_round_0);
- d1 = convolve8_4x4_s16(s1, s2, s3, s4, s5, s6, s7, s8, x_filter_tmp,
+ d1 = convolve8_4x4_s16(s1, s2, s3, s4, s5, s6, s7, s8, x_filter,
horiz_const, shift_round_0);
- d2 = convolve8_4x4_s16(s2, s3, s4, s5, s6, s7, s8, s9, x_filter_tmp,
+ d2 = convolve8_4x4_s16(s2, s3, s4, s5, s6, s7, s8, s9, x_filter,
horiz_const, shift_round_0);
- d3 = convolve8_4x4_s16(s3, s4, s5, s6, s7, s8, s9, s10, x_filter_tmp,
+ d3 = convolve8_4x4_s16(s3, s4, s5, s6, s7, s8, s9, s10, x_filter,
horiz_const, shift_round_0);
transpose_s16_4x4d(&d0, &d1, &d2, &d3);
- if (w == 4) {
- vst1_s16((dst_ptr + 0 * im_dst_stride), d0);
- vst1_s16((dst_ptr + 1 * im_dst_stride), d1);
- vst1_s16((dst_ptr + 2 * im_dst_stride), d2);
- vst1_s16((dst_ptr + 3 * im_dst_stride), d3);
- } else if (w == 2) {
- vst1_lane_u32((uint32_t *)(dst_ptr + 0 * im_dst_stride),
+
+ if (w == 2) {
+ vst1_lane_u32((uint32_t *)(dst_ptr + 0 * dst_stride),
vreinterpret_u32_s16(d0), 0);
- vst1_lane_u32((uint32_t *)(dst_ptr + 1 * im_dst_stride),
+ vst1_lane_u32((uint32_t *)(dst_ptr + 1 * dst_stride),
vreinterpret_u32_s16(d1), 0);
- vst1_lane_u32((uint32_t *)(dst_ptr + 2 * im_dst_stride),
+ vst1_lane_u32((uint32_t *)(dst_ptr + 2 * dst_stride),
vreinterpret_u32_s16(d2), 0);
- vst1_lane_u32((uint32_t *)(dst_ptr + 3 * im_dst_stride),
+ vst1_lane_u32((uint32_t *)(dst_ptr + 3 * dst_stride),
vreinterpret_u32_s16(d3), 0);
+ } else {
+ vst1_s16((dst_ptr + 0 * dst_stride), d0);
+ vst1_s16((dst_ptr + 1 * dst_stride), d1);
+ vst1_s16((dst_ptr + 2 * dst_stride), d2);
+ vst1_s16((dst_ptr + 3 * dst_stride), d3);
}
+
src_ptr += 4 * src_stride;
- dst_ptr += 4 * im_dst_stride;
+ dst_ptr += 4 * dst_stride;
height -= 4;
} while (height >= 4);
if (height) {
assert(height < 4);
- horiz_filter_w4_single_row(src_ptr, src_stride, dst_ptr, im_dst_stride, w,
- height, x_filter_tmp, horiz_const,
- shift_round_0);
+ horiz_filter_w4_single_row(src_ptr, src_stride, dst_ptr, dst_stride, w,
+ height, x_filter, horiz_const, shift_round_0);
}
-#else
- horiz_filter_w4_single_row(src_ptr, src_stride, dst_ptr, im_dst_stride, w,
- height, x_filter_tmp, horiz_const,
- shift_round_0);
-#endif
+
+#else // !defined(__aarch64__)
+ horiz_filter_w4_single_row(src_ptr, src_stride, dst_ptr, dst_stride, w,
+ height, x_filter, horiz_const, shift_round_0);
+#endif // defined(__aarch64__)
} else {
const int16x8_t horiz_const = vdupq_n_s16((1 << (bd + FILTER_BITS - 2)));
- const int16x8_t shift_round_0 = vdupq_n_s16(-(conv_params->round_0 - 1));
+ const int16x8_t shift_round_0 = vdupq_n_s16(-(round_0 - 1));
#if defined(__aarch64__)
- int16_t *d_tmp;
- int16x8_t s0, s1, s2, s3, s4, s5, s6, s7, s8, s9, s10, s11, s12, s13, s14;
- int16x8_t res0, res1, res2, res3, res4, res5, res6, res7;
- do {
- assert(height >= 8);
- __builtin_prefetch(src_ptr + 0 * src_stride);
- __builtin_prefetch(src_ptr + 1 * src_stride);
- __builtin_prefetch(src_ptr + 2 * src_stride);
- __builtin_prefetch(src_ptr + 3 * src_stride);
- __builtin_prefetch(src_ptr + 4 * src_stride);
- __builtin_prefetch(src_ptr + 5 * src_stride);
- __builtin_prefetch(src_ptr + 6 * src_stride);
- __builtin_prefetch(src_ptr + 7 * src_stride);
- load_u8_8x8(src_ptr, src_stride, &t0, &t1, &t2, &t3, &t4, &t5, &t6, &t7);
+ for (; height >= 8; height -= 8) {
+ int16x8_t s0, s1, s2, s3, s4, s5, s6, s7, s8, s9, s10, s11, s12, s13, s14,
+ d0, d1, d2, d3, d4, d5, d6, d7;
+ uint8x8_t t0, t1, t2, t3, t4, t5, t6, t7;
+
+ const uint8_t *s = src_ptr;
+ int16_t *d = dst_ptr;
+ int width = w;
+
+ load_u8_8x8(s, src_stride, &t0, &t1, &t2, &t3, &t4, &t5, &t6, &t7);
transpose_u8_8x8(&t0, &t1, &t2, &t3, &t4, &t5, &t6, &t7);
@@ -1108,18 +1419,7 @@
s5 = vreinterpretq_s16_u16(vmovl_u8(t5));
s6 = vreinterpretq_s16_u16(vmovl_u8(t6));
- width = w;
- s = src_ptr + 7;
- d_tmp = dst_ptr;
-
- __builtin_prefetch(dst_ptr + 0 * im_dst_stride);
- __builtin_prefetch(dst_ptr + 1 * im_dst_stride);
- __builtin_prefetch(dst_ptr + 2 * im_dst_stride);
- __builtin_prefetch(dst_ptr + 3 * im_dst_stride);
- __builtin_prefetch(dst_ptr + 4 * im_dst_stride);
- __builtin_prefetch(dst_ptr + 5 * im_dst_stride);
- __builtin_prefetch(dst_ptr + 6 * im_dst_stride);
- __builtin_prefetch(dst_ptr + 7 * im_dst_stride);
+ s += 7;
do {
load_u8_8x8(s, src_stride, &t0, &t1, &t2, &t3, &t4, &t5, &t6, &t7);
@@ -1135,28 +1435,26 @@
s13 = vreinterpretq_s16_u16(vmovl_u8(t6));
s14 = vreinterpretq_s16_u16(vmovl_u8(t7));
- res0 = convolve8_8x8_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter_tmp,
- horiz_const, shift_round_0);
- res1 = convolve8_8x8_s16(s1, s2, s3, s4, s5, s6, s7, s8, x_filter_tmp,
- horiz_const, shift_round_0);
- res2 = convolve8_8x8_s16(s2, s3, s4, s5, s6, s7, s8, s9, x_filter_tmp,
- horiz_const, shift_round_0);
- res3 = convolve8_8x8_s16(s3, s4, s5, s6, s7, s8, s9, s10, x_filter_tmp,
- horiz_const, shift_round_0);
- res4 = convolve8_8x8_s16(s4, s5, s6, s7, s8, s9, s10, s11, x_filter_tmp,
- horiz_const, shift_round_0);
- res5 = convolve8_8x8_s16(s5, s6, s7, s8, s9, s10, s11, s12,
- x_filter_tmp, horiz_const, shift_round_0);
- res6 = convolve8_8x8_s16(s6, s7, s8, s9, s10, s11, s12, s13,
- x_filter_tmp, horiz_const, shift_round_0);
- res7 = convolve8_8x8_s16(s7, s8, s9, s10, s11, s12, s13, s14,
- x_filter_tmp, horiz_const, shift_round_0);
+ d0 = convolve8_8x8_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter,
+ horiz_const, shift_round_0);
+ d1 = convolve8_8x8_s16(s1, s2, s3, s4, s5, s6, s7, s8, x_filter,
+ horiz_const, shift_round_0);
+ d2 = convolve8_8x8_s16(s2, s3, s4, s5, s6, s7, s8, s9, x_filter,
+ horiz_const, shift_round_0);
+ d3 = convolve8_8x8_s16(s3, s4, s5, s6, s7, s8, s9, s10, x_filter,
+ horiz_const, shift_round_0);
+ d4 = convolve8_8x8_s16(s4, s5, s6, s7, s8, s9, s10, s11, x_filter,
+ horiz_const, shift_round_0);
+ d5 = convolve8_8x8_s16(s5, s6, s7, s8, s9, s10, s11, s12, x_filter,
+ horiz_const, shift_round_0);
+ d6 = convolve8_8x8_s16(s6, s7, s8, s9, s10, s11, s12, s13, x_filter,
+ horiz_const, shift_round_0);
+ d7 = convolve8_8x8_s16(s7, s8, s9, s10, s11, s12, s13, s14, x_filter,
+ horiz_const, shift_round_0);
- transpose_s16_8x8(&res0, &res1, &res2, &res3, &res4, &res5, &res6,
- &res7);
+ transpose_s16_8x8(&d0, &d1, &d2, &d3, &d4, &d5, &d6, &d7);
- store_s16_8x8(d_tmp, im_dst_stride, res0, res1, res2, res3, res4, res5,
- res6, res7);
+ store_s16_8x8(d, dst_stride, d0, d1, d2, d3, d4, d5, d6, d7);
s0 = s8;
s1 = s9;
@@ -1166,248 +1464,262 @@
s5 = s13;
s6 = s14;
s += 8;
- d_tmp += 8;
+ d += 8;
width -= 8;
} while (width > 0);
+
src_ptr += 8 * src_stride;
- dst_ptr += 8 * im_dst_stride;
- height -= 8;
- } while (height >= 8);
+ dst_ptr += 8 * dst_stride;
+ }
- if (height >= 4) {
- assert(height < 8);
- int16x4_t reg0, reg1, reg2, reg3, reg4, reg5, reg6, reg7, reg8, reg9,
- reg10, reg11, reg12, reg13, reg14;
- int16x4_t d0, d1, d2, d3, d4, d5, d6, d7;
- int16x8_t out0, out1, out2, out3;
+ for (; height >= 4; height -= 4) {
+ int16x4_t s0, s1, s2, s3, s4, s5, s6, s7, s8, s9, s10, s11, s12, s13, s14,
+ dd0, dd1, dd2, dd3, dd4, dd5, dd6, dd7;
+ int16x8_t d0, d1, d2, d3;
+ uint8x8_t t0, t1, t2, t3;
- __builtin_prefetch(src_ptr + 0 * src_stride);
- __builtin_prefetch(src_ptr + 1 * src_stride);
- __builtin_prefetch(src_ptr + 2 * src_stride);
- __builtin_prefetch(src_ptr + 3 * src_stride);
+ const uint8_t *s = src_ptr;
+ int16_t *d = dst_ptr;
+ int width = w;
load_u8_8x4(src_ptr, src_stride, &t0, &t1, &t2, &t3);
transpose_u8_8x4(&t0, &t1, &t2, &t3);
- reg0 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t0)));
- reg1 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t1)));
- reg2 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t2)));
- reg3 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t3)));
- reg4 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t0)));
- reg5 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t1)));
- reg6 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t2)));
+ s0 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t0)));
+ s1 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t1)));
+ s2 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t2)));
+ s3 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t3)));
+ s4 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t0)));
+ s5 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t1)));
+ s6 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t2)));
- __builtin_prefetch(dst_ptr + 0 * dst_stride);
- __builtin_prefetch(dst_ptr + 1 * dst_stride);
- __builtin_prefetch(dst_ptr + 2 * dst_stride);
- __builtin_prefetch(dst_ptr + 3 * dst_stride);
-
- s = src_ptr + 7;
- d_tmp = dst_ptr;
- width = w;
+ s += 7;
do {
load_u8_8x4(s, src_stride, &t0, &t1, &t2, &t3);
transpose_u8_8x4(&t0, &t1, &t2, &t3);
- reg7 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t0)));
- reg8 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t1)));
- reg9 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t2)));
- reg10 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t3)));
- reg11 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t0)));
- reg12 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t1)));
- reg13 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t2)));
- reg14 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t3)));
+ s7 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t0)));
+ s8 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t1)));
+ s9 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t2)));
+ s10 = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(t3)));
+ s11 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t0)));
+ s12 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t1)));
+ s13 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t2)));
+ s14 = vget_high_s16(vreinterpretq_s16_u16(vmovl_u8(t3)));
- d0 = convolve8_4x4(reg0, reg1, reg2, reg3, reg4, reg5, reg6, reg7,
- x_filter_tmp);
+ dd0 = convolve8_4x4(s0, s1, s2, s3, s4, s5, s6, s7, x_filter);
+ dd1 = convolve8_4x4(s1, s2, s3, s4, s5, s6, s7, s8, x_filter);
+ dd2 = convolve8_4x4(s2, s3, s4, s5, s6, s7, s8, s9, x_filter);
+ dd3 = convolve8_4x4(s3, s4, s5, s6, s7, s8, s9, s10, x_filter);
+ dd4 = convolve8_4x4(s4, s5, s6, s7, s8, s9, s10, s11, x_filter);
+ dd5 = convolve8_4x4(s5, s6, s7, s8, s9, s10, s11, s12, x_filter);
+ dd6 = convolve8_4x4(s6, s7, s8, s9, s10, s11, s12, s13, x_filter);
+ dd7 = convolve8_4x4(s7, s8, s9, s10, s11, s12, s13, s14, x_filter);
- d1 = convolve8_4x4(reg1, reg2, reg3, reg4, reg5, reg6, reg7, reg8,
- x_filter_tmp);
+ transpose_s16_4x8(&dd0, &dd1, &dd2, &dd3, &dd4, &dd5, &dd6, &dd7, &d0,
+ &d1, &d2, &d3);
- d2 = convolve8_4x4(reg2, reg3, reg4, reg5, reg6, reg7, reg8, reg9,
- x_filter_tmp);
+ d0 = vaddq_s16(d0, horiz_const);
+ d1 = vaddq_s16(d1, horiz_const);
+ d2 = vaddq_s16(d2, horiz_const);
+ d3 = vaddq_s16(d3, horiz_const);
- d3 = convolve8_4x4(reg3, reg4, reg5, reg6, reg7, reg8, reg9, reg10,
- x_filter_tmp);
+ d0 = vqrshlq_s16(d0, shift_round_0);
+ d1 = vqrshlq_s16(d1, shift_round_0);
+ d2 = vqrshlq_s16(d2, shift_round_0);
+ d3 = vqrshlq_s16(d3, shift_round_0);
- d4 = convolve8_4x4(reg4, reg5, reg6, reg7, reg8, reg9, reg10, reg11,
- x_filter_tmp);
+ store_s16_8x4(d, dst_stride, d0, d1, d2, d3);
- d5 = convolve8_4x4(reg5, reg6, reg7, reg8, reg9, reg10, reg11, reg12,
- x_filter_tmp);
-
- d6 = convolve8_4x4(reg6, reg7, reg8, reg9, reg10, reg11, reg12, reg13,
- x_filter_tmp);
-
- d7 = convolve8_4x4(reg7, reg8, reg9, reg10, reg11, reg12, reg13, reg14,
- x_filter_tmp);
-
- transpose_s16_4x8(&d0, &d1, &d2, &d3, &d4, &d5, &d6, &d7, &out0, &out1,
- &out2, &out3);
-
- out0 = vaddq_s16(out0, horiz_const);
- out0 = vqrshlq_s16(out0, shift_round_0);
-
- out1 = vaddq_s16(out1, horiz_const);
- out1 = vqrshlq_s16(out1, shift_round_0);
-
- out2 = vaddq_s16(out2, horiz_const);
- out2 = vqrshlq_s16(out2, shift_round_0);
-
- out3 = vaddq_s16(out3, horiz_const);
- out3 = vqrshlq_s16(out3, shift_round_0);
-
- store_s16_8x4(d_tmp, im_dst_stride, out0, out1, out2, out3);
-
- reg0 = reg8;
- reg1 = reg9;
- reg2 = reg10;
- reg3 = reg11;
- reg4 = reg12;
- reg5 = reg13;
- reg6 = reg14;
+ s0 = s8;
+ s1 = s9;
+ s2 = s10;
+ s3 = s11;
+ s4 = s12;
+ s5 = s13;
+ s6 = s14;
s += 8;
- d_tmp += 8;
+ d += 8;
width -= 8;
} while (width > 0);
+
src_ptr += 4 * src_stride;
- dst_ptr += 4 * im_dst_stride;
- height -= 4;
+ dst_ptr += 4 * dst_stride;
}
if (height) {
assert(height < 4);
- horiz_filter_w8_single_row(src_ptr, src_stride, dst_ptr, im_stride, w,
- height, x_filter_tmp, horiz_const,
- shift_round_0);
+ horiz_filter_w8_single_row(src_ptr, src_stride, dst_ptr, dst_stride, w,
+ height, x_filter, horiz_const, shift_round_0);
}
-#else
- horiz_filter_w8_single_row(src_ptr, src_stride, dst_ptr, im_stride, w,
- height, x_filter_tmp, horiz_const,
- shift_round_0);
-#endif
+#else // !defined(__aarch64__)
+ horiz_filter_w8_single_row(src_ptr, src_stride, dst_ptr, dst_stride, w,
+ height, x_filter, horiz_const, shift_round_0);
+#endif // defined(__aarch64__)
}
+}
- // vertical
- {
- uint8_t *dst_u8_ptr, *d_u8;
- int16_t *v_src_ptr, *v_s;
+#endif // defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
- const int32_t sub_const = (1 << (offset_bits - conv_params->round_1)) +
- (1 << (offset_bits - conv_params->round_1 - 1));
- const int16_t *y_filter = av1_get_interp_filter_subpel_kernel(
- filter_params_y, subpel_y_qn & SUBPEL_MASK);
+static INLINE void av1_convolve_2d_sr_vert_neon(
+ int16_t *src_ptr, int src_stride, uint8_t *dst_ptr, int dst_stride, int w,
+ int h, const int16x8_t y_filter, ConvolveParams *conv_params) {
+ const int bd = 8;
+ const int16_t round_bits =
+ FILTER_BITS * 2 - conv_params->round_0 - conv_params->round_1;
+ const int16x8_t vec_round_bits = vdupq_n_s16(-round_bits);
+ const int offset_bits = bd + 2 * FILTER_BITS - conv_params->round_0;
- const int32x4_t round_shift_vec = vdupq_n_s32(-(conv_params->round_1));
- const int32x4_t offset_const = vdupq_n_s32(1 << offset_bits);
- const int32x4_t sub_const_vec = vdupq_n_s32(sub_const);
+ const int32_t sub_const = (1 << (offset_bits - conv_params->round_1)) +
+ (1 << (offset_bits - conv_params->round_1 - 1));
- src_stride = im_stride;
- v_src_ptr = im_block;
- dst_u8_ptr = dst;
+ const int32x4_t round_shift_vec = vdupq_n_s32(-(conv_params->round_1));
+ const int32x4_t offset_const = vdupq_n_s32(1 << offset_bits);
+ const int32x4_t sub_const_vec = vdupq_n_s32(sub_const);
- height = h;
- width = w;
-
- if (width <= 4) {
- int16x4_t s0, s1, s2, s3, s4, s5, s6, s7;
- uint16x4_t d0;
- uint16x8_t dd0;
- uint8x8_t d01;
+ if (w <= 4) {
+ int16x4_t s0, s1, s2, s3, s4, s5, s6, s7, d0;
+ int16x8_t dd0;
+ uint8x8_t d01;
#if defined(__aarch64__)
- int16x4_t s8, s9, s10;
- uint16x4_t d1, d2, d3;
- uint16x8_t dd1;
- uint8x8_t d23;
-#endif
+ int16x4_t s8, s9, s10, d1, d2, d3;
+ int16x8_t dd1;
+ uint8x8_t d23;
+#endif // defined(__aarch64__)
- d_u8 = dst_u8_ptr;
- v_s = v_src_ptr;
+ int16_t *s = src_ptr;
+ uint8_t *d = dst_ptr;
- __builtin_prefetch(v_s + 0 * im_stride);
- __builtin_prefetch(v_s + 1 * im_stride);
- __builtin_prefetch(v_s + 2 * im_stride);
- __builtin_prefetch(v_s + 3 * im_stride);
- __builtin_prefetch(v_s + 4 * im_stride);
- __builtin_prefetch(v_s + 5 * im_stride);
- __builtin_prefetch(v_s + 6 * im_stride);
- __builtin_prefetch(v_s + 7 * im_stride);
+ load_s16_4x8(s, src_stride, &s0, &s1, &s2, &s3, &s4, &s5, &s6, &s7);
+ s += (7 * src_stride);
- load_s16_4x8(v_s, im_stride, &s0, &s1, &s2, &s3, &s4, &s5, &s6, &s7);
- v_s += (7 * im_stride);
+ do {
+#if defined(__aarch64__)
+ load_s16_4x4(s, src_stride, &s7, &s8, &s9, &s10);
+ s += (4 * src_stride);
+
+ d0 = convolve8_vert_4x4_s32(s0, s1, s2, s3, s4, s5, s6, s7, y_filter,
+ round_shift_vec, offset_const, sub_const_vec);
+ d1 = convolve8_vert_4x4_s32(s1, s2, s3, s4, s5, s6, s7, s8, y_filter,
+ round_shift_vec, offset_const, sub_const_vec);
+ d2 = convolve8_vert_4x4_s32(s2, s3, s4, s5, s6, s7, s8, s9, y_filter,
+ round_shift_vec, offset_const, sub_const_vec);
+ d3 = convolve8_vert_4x4_s32(s3, s4, s5, s6, s7, s8, s9, s10, y_filter,
+ round_shift_vec, offset_const, sub_const_vec);
+
+ dd0 = vqrshlq_s16(vcombine_s16(d0, d1), vec_round_bits);
+ dd1 = vqrshlq_s16(vcombine_s16(d2, d3), vec_round_bits);
+
+ d01 = vqmovun_s16(dd0);
+ d23 = vqmovun_s16(dd1);
+
+ if (w == 4) {
+ vst1_lane_u32((uint32_t *)d, vreinterpret_u32_u8(d01), 0);
+ d += dst_stride;
+ vst1_lane_u32((uint32_t *)d, vreinterpret_u32_u8(d01), 1);
+ d += dst_stride;
+ if (h != 2) {
+ vst1_lane_u32((uint32_t *)d, vreinterpret_u32_u8(d23), 0);
+ d += dst_stride;
+ vst1_lane_u32((uint32_t *)d, vreinterpret_u32_u8(d23), 1);
+ d += dst_stride;
+ }
+ } else {
+ vst1_lane_u16((uint16_t *)d, vreinterpret_u16_u8(d01), 0);
+ d += dst_stride;
+ vst1_lane_u16((uint16_t *)d, vreinterpret_u16_u8(d01), 2);
+ d += dst_stride;
+ if (h != 2) {
+ vst1_lane_u16((uint16_t *)d, vreinterpret_u16_u8(d23), 0);
+ d += dst_stride;
+ vst1_lane_u16((uint16_t *)d, vreinterpret_u16_u8(d23), 2);
+ d += dst_stride;
+ }
+ }
+
+ s0 = s4;
+ s1 = s5;
+ s2 = s6;
+ s3 = s7;
+ s4 = s8;
+ s5 = s9;
+ s6 = s10;
+ h -= 4;
+#else // !defined(__aarch64__)
+ s7 = vld1_s16(s);
+ s += src_stride;
+
+ d0 = convolve8_vert_4x4_s32(s0, s1, s2, s3, s4, s5, s6, s7, y_filter,
+ round_shift_vec, offset_const, sub_const_vec);
+
+ dd0 = vqrshlq_s16(vcombine_s16(d0, d0), vec_round_bits);
+ d01 = vqmovun_s16(dd0);
+
+ if (w == 2) {
+ vst1_lane_u16((uint16_t *)d, vreinterpret_u16_u8(d01), 0);
+ d += dst_stride;
+ } else {
+ vst1_lane_u32((uint32_t *)d, vreinterpret_u32_u8(d01), 0);
+ d += dst_stride;
+ }
+
+ s0 = s1;
+ s1 = s2;
+ s2 = s3;
+ s3 = s4;
+ s4 = s5;
+ s5 = s6;
+ s6 = s7;
+ h--;
+#endif // defined(__aarch64__)
+ } while (h > 0);
+ } else {
+ // if width is a multiple of 8 & height is a multiple of 4
+ int16x8_t s0, s1, s2, s3, s4, s5, s6, s7;
+ uint8x8_t d0;
+#if defined(__aarch64__)
+ int16x8_t s8, s9, s10;
+ uint8x8_t d1, d2, d3;
+#endif // defined(__aarch64__)
+
+ do {
+ int height = h;
+ int16_t *s = src_ptr;
+ uint8_t *d = dst_ptr;
+
+ load_s16_8x8(s, src_stride, &s0, &s1, &s2, &s3, &s4, &s5, &s6, &s7);
+ s += (7 * src_stride);
do {
#if defined(__aarch64__)
- load_s16_4x4(v_s, im_stride, &s7, &s8, &s9, &s10);
- v_s += (im_stride << 2);
+ load_s16_8x4(s, src_stride, &s7, &s8, &s9, &s10);
+ s += (4 * src_stride);
- __builtin_prefetch(d_u8 + 0 * dst_stride);
- __builtin_prefetch(d_u8 + 1 * dst_stride);
- __builtin_prefetch(d_u8 + 2 * dst_stride);
- __builtin_prefetch(d_u8 + 3 * dst_stride);
-
- d0 = convolve8_vert_4x4_s32(s0, s1, s2, s3, s4, s5, s6, s7, y_filter,
+ d0 = convolve8_vert_8x4_s32(s0, s1, s2, s3, s4, s5, s6, s7, y_filter,
round_shift_vec, offset_const,
- sub_const_vec);
- d1 = convolve8_vert_4x4_s32(s1, s2, s3, s4, s5, s6, s7, s8, y_filter,
+ sub_const_vec, vec_round_bits);
+ d1 = convolve8_vert_8x4_s32(s1, s2, s3, s4, s5, s6, s7, s8, y_filter,
round_shift_vec, offset_const,
- sub_const_vec);
- d2 = convolve8_vert_4x4_s32(s2, s3, s4, s5, s6, s7, s8, s9, y_filter,
+ sub_const_vec, vec_round_bits);
+ d2 = convolve8_vert_8x4_s32(s2, s3, s4, s5, s6, s7, s8, s9, y_filter,
round_shift_vec, offset_const,
- sub_const_vec);
- d3 = convolve8_vert_4x4_s32(s3, s4, s5, s6, s7, s8, s9, s10, y_filter,
+ sub_const_vec, vec_round_bits);
+ d3 = convolve8_vert_8x4_s32(s3, s4, s5, s6, s7, s8, s9, s10, y_filter,
round_shift_vec, offset_const,
- sub_const_vec);
+ sub_const_vec, vec_round_bits);
- dd0 = vqrshlq_u16(vcombine_u16(d0, d1), vec_round_bits);
- dd1 = vqrshlq_u16(vcombine_u16(d2, d3), vec_round_bits);
-
- d01 = vqmovn_u16(dd0);
- d23 = vqmovn_u16(dd1);
-
- if ((w == 4) && (h != 2)) {
- vst1_lane_u32((uint32_t *)d_u8, vreinterpret_u32_u8(d01),
- 0); // 00 01 02 03
- d_u8 += dst_stride;
- vst1_lane_u32((uint32_t *)d_u8, vreinterpret_u32_u8(d01),
- 1); // 10 11 12 13
- d_u8 += dst_stride;
- vst1_lane_u32((uint32_t *)d_u8, vreinterpret_u32_u8(d23),
- 0); // 20 21 22 23
- d_u8 += dst_stride;
- vst1_lane_u32((uint32_t *)d_u8, vreinterpret_u32_u8(d23),
- 1); // 30 31 32 33
- d_u8 += dst_stride;
- } else if ((w == 2) && (h != 2)) {
- vst1_lane_u16((uint16_t *)d_u8, vreinterpret_u16_u8(d01),
- 0); // 00 01
- d_u8 += dst_stride;
- vst1_lane_u16((uint16_t *)d_u8, vreinterpret_u16_u8(d01),
- 2); // 10 11
- d_u8 += dst_stride;
- vst1_lane_u16((uint16_t *)d_u8, vreinterpret_u16_u8(d23),
- 0); // 20 21
- d_u8 += dst_stride;
- vst1_lane_u16((uint16_t *)d_u8, vreinterpret_u16_u8(d23),
- 2); // 30 31
- d_u8 += dst_stride;
- } else if ((w == 4) && (h == 2)) {
- vst1_lane_u32((uint32_t *)d_u8, vreinterpret_u32_u8(d01),
- 0); // 00 01 02 03
- d_u8 += dst_stride;
- vst1_lane_u32((uint32_t *)d_u8, vreinterpret_u32_u8(d01),
- 1); // 10 11 12 13
- d_u8 += dst_stride;
- } else if ((w == 2) && (h == 2)) {
- vst1_lane_u16((uint16_t *)d_u8, vreinterpret_u16_u8(d01),
- 0); // 00 01
- d_u8 += dst_stride;
- vst1_lane_u16((uint16_t *)d_u8, vreinterpret_u16_u8(d01),
- 2); // 10 11
- d_u8 += dst_stride;
+ vst1_u8(d, d0);
+ d += dst_stride;
+ vst1_u8(d, d1);
+ d += dst_stride;
+ if (h != 2) {
+ vst1_u8(d, d2);
+ d += dst_stride;
+ vst1_u8(d, d3);
+ d += dst_stride;
}
s0 = s4;
@@ -1418,29 +1730,16 @@
s5 = s9;
s6 = s10;
height -= 4;
-#else
- s7 = vld1_s16(v_s);
- v_s += im_stride;
+#else // !defined(__aarch64__)
+ s7 = vld1q_s16(s);
+ s += src_stride;
- __builtin_prefetch(d_u8 + 0 * dst_stride);
-
- d0 = convolve8_vert_4x4_s32(s0, s1, s2, s3, s4, s5, s6, s7, y_filter,
+ d0 = convolve8_vert_8x4_s32(s0, s1, s2, s3, s4, s5, s6, s7, y_filter,
round_shift_vec, offset_const,
- sub_const_vec);
+ sub_const_vec, vec_round_bits);
- dd0 = vqrshlq_u16(vcombine_u16(d0, d0), vec_round_bits);
- d01 = vqmovn_u16(dd0);
-
- if (w == 4) {
- vst1_lane_u32((uint32_t *)d_u8, vreinterpret_u32_u8(d01),
- 0); // 00 01 02 03
- d_u8 += dst_stride;
-
- } else if (w == 2) {
- vst1_lane_u16((uint16_t *)d_u8, vreinterpret_u16_u8(d01),
- 0); // 00 01
- d_u8 += dst_stride;
- }
+ vst1_u8(d, d0);
+ d += dst_stride;
s0 = s1;
s1 = s2;
@@ -1449,109 +1748,49 @@
s4 = s5;
s5 = s6;
s6 = s7;
- height -= 1;
-#endif
+ height--;
+#endif // defined(__aarch64__)
} while (height > 0);
- } else {
- // if width is a multiple of 8 & height is a multiple of 4
- int16x8_t s0, s1, s2, s3, s4, s5, s6, s7;
- uint8x8_t res0;
-#if defined(__aarch64__)
- int16x8_t s8, s9, s10;
- uint8x8_t res1, res2, res3;
-#endif
- do {
- __builtin_prefetch(v_src_ptr + 0 * im_stride);
- __builtin_prefetch(v_src_ptr + 1 * im_stride);
- __builtin_prefetch(v_src_ptr + 2 * im_stride);
- __builtin_prefetch(v_src_ptr + 3 * im_stride);
- __builtin_prefetch(v_src_ptr + 4 * im_stride);
- __builtin_prefetch(v_src_ptr + 5 * im_stride);
- __builtin_prefetch(v_src_ptr + 6 * im_stride);
- __builtin_prefetch(v_src_ptr + 7 * im_stride);
+ src_ptr += 8;
+ dst_ptr += 8;
+ w -= 8;
+ } while (w > 0);
+ }
+}
- v_s = v_src_ptr;
- load_s16_8x8(v_s, im_stride, &s0, &s1, &s2, &s3, &s4, &s5, &s6, &s7);
- v_s += (7 * im_stride);
+void av1_convolve_2d_sr_neon(const uint8_t *src, int src_stride, uint8_t *dst,
+ int dst_stride, int w, int h,
+ const InterpFilterParams *filter_params_x,
+ const InterpFilterParams *filter_params_y,
+ const int subpel_x_qn, const int subpel_y_qn,
+ ConvolveParams *conv_params) {
+ if (filter_params_x->taps > 8) {
+ av1_convolve_2d_sr_c(src, src_stride, dst, dst_stride, w, h,
+ filter_params_x, filter_params_y, subpel_x_qn,
+ subpel_y_qn, conv_params);
+ } else {
+ DECLARE_ALIGNED(16, int16_t,
+ im_block[(MAX_SB_SIZE + HORIZ_EXTRA_ROWS) * MAX_SB_SIZE]);
- d_u8 = dst_u8_ptr;
- height = h;
+ const int im_h = h + filter_params_y->taps - 1;
+ const int im_stride = MAX_SB_SIZE;
+ const int vert_offset = filter_params_y->taps / 2 - 1;
+ const int horiz_offset = filter_params_x->taps / 2 - 1;
+ const uint8_t *src_ptr = src - vert_offset * src_stride - horiz_offset;
- do {
-#if defined(__aarch64__)
- load_s16_8x4(v_s, im_stride, &s7, &s8, &s9, &s10);
- v_s += (im_stride << 2);
+ const int16_t *x_filter_ptr = av1_get_interp_filter_subpel_kernel(
+ filter_params_x, subpel_x_qn & SUBPEL_MASK);
+ const int16_t *y_filter_ptr = av1_get_interp_filter_subpel_kernel(
+ filter_params_y, subpel_y_qn & SUBPEL_MASK);
+ const int16x8_t x_filter = vld1q_s16(x_filter_ptr);
+ const int16x8_t y_filter = vld1q_s16(y_filter_ptr);
- __builtin_prefetch(d_u8 + 4 * dst_stride);
- __builtin_prefetch(d_u8 + 5 * dst_stride);
- __builtin_prefetch(d_u8 + 6 * dst_stride);
- __builtin_prefetch(d_u8 + 7 * dst_stride);
+ av1_convolve_2d_sr_horiz_neon(src_ptr, src_stride, im_block, im_stride, w,
+ im_h, x_filter, conv_params->round_0);
- res0 = convolve8_vert_8x4_s32(s0, s1, s2, s3, s4, s5, s6, s7,
- y_filter, round_shift_vec, offset_const,
- sub_const_vec, vec_round_bits);
- res1 = convolve8_vert_8x4_s32(s1, s2, s3, s4, s5, s6, s7, s8,
- y_filter, round_shift_vec, offset_const,
- sub_const_vec, vec_round_bits);
- res2 = convolve8_vert_8x4_s32(s2, s3, s4, s5, s6, s7, s8, s9,
- y_filter, round_shift_vec, offset_const,
- sub_const_vec, vec_round_bits);
- res3 = convolve8_vert_8x4_s32(s3, s4, s5, s6, s7, s8, s9, s10,
- y_filter, round_shift_vec, offset_const,
- sub_const_vec, vec_round_bits);
-
- if (h != 2) {
- vst1_u8(d_u8, res0);
- d_u8 += dst_stride;
- vst1_u8(d_u8, res1);
- d_u8 += dst_stride;
- vst1_u8(d_u8, res2);
- d_u8 += dst_stride;
- vst1_u8(d_u8, res3);
- d_u8 += dst_stride;
- } else {
- vst1_u8(d_u8, res0);
- d_u8 += dst_stride;
- vst1_u8(d_u8, res1);
- d_u8 += dst_stride;
- }
- s0 = s4;
- s1 = s5;
- s2 = s6;
- s3 = s7;
- s4 = s8;
- s5 = s9;
- s6 = s10;
- height -= 4;
-#else
- s7 = vld1q_s16(v_s);
- v_s += im_stride;
-
- __builtin_prefetch(d_u8 + 0 * dst_stride);
-
- res0 = convolve8_vert_8x4_s32(s0, s1, s2, s3, s4, s5, s6, s7,
- y_filter, round_shift_vec, offset_const,
- sub_const_vec, vec_round_bits);
-
- vst1_u8(d_u8, res0);
- d_u8 += dst_stride;
-
- s0 = s1;
- s1 = s2;
- s2 = s3;
- s3 = s4;
- s4 = s5;
- s5 = s6;
- s6 = s7;
- height -= 1;
-#endif
- } while (height > 0);
- v_src_ptr += 8;
- dst_u8_ptr += 8;
- w -= 8;
- } while (w > 0);
- }
+ av1_convolve_2d_sr_vert_neon(im_block, im_stride, dst, dst_stride, w, h,
+ y_filter, conv_params);
}
}
@@ -1574,8 +1813,6 @@
const uint8_t *const src_x = &src[x_q4 >> SUBPEL_BITS];
if (x_q4 & SUBPEL_MASK) {
const int16x8_t filters = vld1q_s16(x_filters[x_q4 & SUBPEL_MASK]);
- const int16x4_t filter3 = vdup_lane_s16(vget_low_s16(filters), 3);
- const int16x4_t filter4 = vdup_lane_s16(vget_high_s16(filters), 0);
uint8x8_t s[8], d;
int16x8_t ss[4];
int16x4_t t[8], tt;
@@ -1597,7 +1834,7 @@
t[7] = vget_high_s16(ss[3]);
tt = convolve8_4(t[0], t[1], t[2], t[3], t[4], t[5], t[6], t[7],
- filters, filter3, filter4);
+ filters);
d = vqrshrun_n_s16(vcombine_s16(tt, tt), 7);
vst1_lane_u32((uint32_t *)&temp[4 * z], vreinterpret_u32_u8(d), 0);
} else {
@@ -1703,8 +1940,6 @@
if (y_q4 & SUBPEL_MASK) {
const int16x8_t filters = vld1q_s16(y_filters[y_q4 & SUBPEL_MASK]);
- const int16x4_t filter3 = vdup_lane_s16(vget_low_s16(filters), 3);
- const int16x4_t filter4 = vdup_lane_s16(vget_high_s16(filters), 0);
uint8x8_t s[8], d;
int16x4_t t[8], tt;
@@ -1719,8 +1954,7 @@
t[6] = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(s[6])));
t[7] = vget_low_s16(vreinterpretq_s16_u16(vmovl_u8(s[7])));
- tt = convolve8_4(t[0], t[1], t[2], t[3], t[4], t[5], t[6], t[7], filters,
- filter3, filter4);
+ tt = convolve8_4(t[0], t[1], t[2], t[3], t[4], t[5], t[6], t[7], filters);
d = vqrshrun_n_s16(vcombine_s16(tt, tt), 7);
vst1_lane_u32((uint32_t *)dst, vreinterpret_u32_u8(d), 0);
} else {
diff --git a/av1/common/arm/convolve_neon.h b/av1/common/arm/convolve_neon.h
index 27a996c..e8c2ad6 100644
--- a/av1/common/arm/convolve_neon.h
+++ b/av1/common/arm/convolve_neon.h
@@ -19,21 +19,19 @@
const int16x4_t s2, const int16x4_t s3,
const int16x4_t s4, const int16x4_t s5,
const int16x4_t s6, const int16x4_t s7,
- const int16x8_t filters,
- const int16x4_t filter3,
- const int16x4_t filter4) {
- const int16x4_t filters_lo = vget_low_s16(filters);
- const int16x4_t filters_hi = vget_high_s16(filters);
+ const int16x8_t filter) {
+ const int16x4_t filter_lo = vget_low_s16(filter);
+ const int16x4_t filter_hi = vget_high_s16(filter);
int16x4_t sum;
- sum = vmul_lane_s16(s0, filters_lo, 0);
- sum = vmla_lane_s16(sum, s1, filters_lo, 1);
- sum = vmla_lane_s16(sum, s2, filters_lo, 2);
- sum = vmla_lane_s16(sum, s5, filters_hi, 1);
- sum = vmla_lane_s16(sum, s6, filters_hi, 2);
- sum = vmla_lane_s16(sum, s7, filters_hi, 3);
- sum = vqadd_s16(sum, vmul_s16(s3, filter3));
- sum = vqadd_s16(sum, vmul_s16(s4, filter4));
+ sum = vmul_lane_s16(s0, filter_lo, 0);
+ sum = vmla_lane_s16(sum, s1, filter_lo, 1);
+ sum = vmla_lane_s16(sum, s2, filter_lo, 2);
+ sum = vmla_lane_s16(sum, s5, filter_hi, 1);
+ sum = vmla_lane_s16(sum, s6, filter_hi, 2);
+ sum = vmla_lane_s16(sum, s7, filter_hi, 3);
+ sum = vqadd_s16(sum, vmul_lane_s16(s3, filter_lo, 3));
+ sum = vqadd_s16(sum, vmul_lane_s16(s4, filter_hi, 0));
return sum;
}
@@ -41,28 +39,24 @@
const int16x8_t s2, const int16x8_t s3,
const int16x8_t s4, const int16x8_t s5,
const int16x8_t s6, const int16x8_t s7,
- const int16x8_t filters,
- const int16x8_t filter3,
- const int16x8_t filter4) {
- const int16x4_t filters_lo = vget_low_s16(filters);
- const int16x4_t filters_hi = vget_high_s16(filters);
+ const int16x8_t filter) {
+ const int16x4_t filter_lo = vget_low_s16(filter);
+ const int16x4_t filter_hi = vget_high_s16(filter);
int16x8_t sum;
- sum = vmulq_lane_s16(s0, filters_lo, 0);
- sum = vmlaq_lane_s16(sum, s1, filters_lo, 1);
- sum = vmlaq_lane_s16(sum, s2, filters_lo, 2);
- sum = vmlaq_lane_s16(sum, s5, filters_hi, 1);
- sum = vmlaq_lane_s16(sum, s6, filters_hi, 2);
- sum = vmlaq_lane_s16(sum, s7, filters_hi, 3);
- sum = vqaddq_s16(sum, vmulq_s16(s3, filter3));
- sum = vqaddq_s16(sum, vmulq_s16(s4, filter4));
+ sum = vmulq_lane_s16(s0, filter_lo, 0);
+ sum = vmlaq_lane_s16(sum, s1, filter_lo, 1);
+ sum = vmlaq_lane_s16(sum, s2, filter_lo, 2);
+ sum = vmlaq_lane_s16(sum, s5, filter_hi, 1);
+ sum = vmlaq_lane_s16(sum, s6, filter_hi, 2);
+ sum = vmlaq_lane_s16(sum, s7, filter_hi, 3);
+ sum = vqaddq_s16(sum, vmulq_lane_s16(s3, filter_lo, 3));
+ sum = vqaddq_s16(sum, vmulq_lane_s16(s4, filter_hi, 0));
return vqrshrun_n_s16(sum, 7);
}
static INLINE uint8x8_t scale_filter_8(const uint8x8_t *const s,
- const int16x8_t filters) {
- const int16x8_t filter3 = vdupq_lane_s16(vget_low_s16(filters), 3);
- const int16x8_t filter4 = vdupq_lane_s16(vget_high_s16(filters), 0);
+ const int16x8_t filter) {
int16x8_t ss[8];
ss[0] = vreinterpretq_s16_u16(vmovl_u8(s[0]));
@@ -75,7 +69,7 @@
ss[7] = vreinterpretq_s16_u16(vmovl_u8(s[7]));
return convolve8_8(ss[0], ss[1], ss[2], ss[3], ss[4], ss[5], ss[6], ss[7],
- filters, filter3, filter4);
+ filter);
}
static INLINE uint8x8_t wiener_convolve8_vert_4x8(
@@ -85,28 +79,27 @@
const int round1_bits) {
int16x8_t ss0, ss1, ss2;
int32x4_t sum0, sum1;
- uint16x4_t tmp0, tmp1;
- uint16x8_t tmp;
+ int16x8_t tmp;
uint8x8_t res;
const int32_t round_const = (1 << (bd + round1_bits - 1));
const int32x4_t round_bits = vdupq_n_s32(-round1_bits);
- const int32x4_t zero = vdupq_n_s32(0);
const int32x4_t round_vec = vdupq_n_s32(round_const);
+ const int16x4_t filter = vld1_s16(filter_y);
ss0 = vaddq_s16(s0, s6);
ss1 = vaddq_s16(s1, s5);
ss2 = vaddq_s16(s2, s4);
- sum0 = vmull_n_s16(vget_low_s16(ss0), filter_y[0]);
- sum0 = vmlal_n_s16(sum0, vget_low_s16(ss1), filter_y[1]);
- sum0 = vmlal_n_s16(sum0, vget_low_s16(ss2), filter_y[2]);
- sum0 = vmlal_n_s16(sum0, vget_low_s16(s3), filter_y[3]);
+ sum0 = vmull_lane_s16(vget_low_s16(ss0), filter, 0);
+ sum0 = vmlal_lane_s16(sum0, vget_low_s16(ss1), filter, 1);
+ sum0 = vmlal_lane_s16(sum0, vget_low_s16(ss2), filter, 2);
+ sum0 = vmlal_lane_s16(sum0, vget_low_s16(s3), filter, 3);
- sum1 = vmull_n_s16(vget_high_s16(ss0), filter_y[0]);
- sum1 = vmlal_n_s16(sum1, vget_high_s16(ss1), filter_y[1]);
- sum1 = vmlal_n_s16(sum1, vget_high_s16(ss2), filter_y[2]);
- sum1 = vmlal_n_s16(sum1, vget_high_s16(s3), filter_y[3]);
+ sum1 = vmull_lane_s16(vget_high_s16(ss0), filter, 0);
+ sum1 = vmlal_lane_s16(sum1, vget_high_s16(ss1), filter, 1);
+ sum1 = vmlal_lane_s16(sum1, vget_high_s16(ss2), filter, 2);
+ sum1 = vmlal_lane_s16(sum1, vget_high_s16(s3), filter, 3);
sum0 = vsubq_s32(sum0, round_vec);
sum1 = vsubq_s32(sum1, round_vec);
@@ -115,14 +108,9 @@
sum0 = vrshlq_s32(sum0, round_bits);
sum1 = vrshlq_s32(sum1, round_bits);
- sum0 = vmaxq_s32(sum0, zero);
- sum1 = vmaxq_s32(sum1, zero);
-
/* from int32x4_t to uint8x8_t */
- tmp0 = vqmovn_u32(vreinterpretq_u32_s32(sum0));
- tmp1 = vqmovn_u32(vreinterpretq_u32_s32(sum1));
- tmp = vcombine_u16(tmp0, tmp1);
- res = vqmovn_u16(tmp);
+ tmp = vcombine_s16(vmovn_s32(sum0), vmovn_s32(sum1));
+ res = vqmovun_s16(tmp);
return res;
}
@@ -143,10 +131,11 @@
const int32x4_t round_vec_0 = vdupq_n_s32(round_const_0);
const int32x4_t round_vec_1 = vdupq_n_s32(round_const_1);
+ const int16x4_t filter = vld1_s16(filter_x);
- sum = vmulq_n_s16(s0, filter_x[0]);
- sum = vmlaq_n_s16(sum, s1, filter_x[1]);
- sum = vmlaq_n_s16(sum, s2, filter_x[2]);
+ sum = vmulq_lane_s16(s0, filter, 0);
+ sum = vmlaq_lane_s16(sum, s1, filter, 1);
+ sum = vmlaq_lane_s16(sum, s2, filter, 2);
/* sum from 16x8 to 2 32x4 registers */
sum_0 = vmovl_s16(vget_low_s16(sum));
@@ -156,8 +145,8 @@
* then max value possible = 128*128*255 exceeding 16 bit
*/
- s3_0 = vmull_n_s16(vget_low_s16(s3), filter_x[3]);
- s3_1 = vmull_n_s16(vget_high_s16(s3), filter_x[3]);
+ s3_0 = vmull_lane_s16(vget_low_s16(s3), filter, 3);
+ s3_1 = vmull_lane_s16(vget_high_s16(s3), filter, 3);
sum_0 = vaddq_s32(sum_0, s3_0);
sum_1 = vaddq_s32(sum_1, s3_1);
@@ -189,73 +178,145 @@
const int32_t round_const_0 = (1 << (bd + FILTER_BITS - 1));
const int32_t round_const_1 = (1 << (bd + 1 + FILTER_BITS - round0_bits)) - 1;
const int32x4_t round_bits = vdupq_n_s32(-round0_bits);
- const int32x4_t zero = vdupq_n_s32(0);
const int32x4_t round_vec_0 = vdupq_n_s32(round_const_0);
const int32x4_t round_vec_1 = vdupq_n_s32(round_const_1);
+ const int16x4_t filter = vld1_s16(filter_x);
temp0 = vadd_s16(s0, s6);
temp1 = vadd_s16(s1, s5);
temp2 = vadd_s16(s2, s4);
- sum = vmul_n_s16(temp0, filter_x[0]);
- sum = vmla_n_s16(sum, temp1, filter_x[1]);
- sum = vmla_n_s16(sum, temp2, filter_x[2]);
+ sum = vmul_lane_s16(temp0, filter, 0);
+ sum = vmla_lane_s16(sum, temp1, filter, 1);
+ sum = vmla_lane_s16(sum, temp2, filter, 2);
sum_0 = vmovl_s16(sum);
/* s[3]*128 -- and filter coff max can be 128.
* then max value possible = 128*128*255 Therefore, 32 bits are required to
* hold the result.
*/
- s3_0 = vmull_n_s16(s3, filter_x[3]);
+ s3_0 = vmull_lane_s16(s3, filter, 3);
sum_0 = vaddq_s32(sum_0, s3_0);
sum_0 = vaddq_s32(sum_0, round_vec_0);
sum_0 = vrshlq_s32(sum_0, round_bits);
- sum_0 = vmaxq_s32(sum_0, zero);
sum_0 = vminq_s32(sum_0, round_vec_1);
res = vqmovun_s32(sum_0);
return res;
}
-static INLINE int16x8_t
-convolve8_8x8_s16(const int16x8_t s0, const int16x8_t s1, const int16x8_t s2,
- const int16x8_t s3, const int16x8_t s4, const int16x8_t s5,
- const int16x8_t s6, const int16x8_t s7, const int16_t *filter,
- const int16x8_t horiz_const, const int16x8_t shift_round_0) {
+static INLINE int16x8_t convolve8_8x8_s16(
+ const int16x8_t s0, const int16x8_t s1, const int16x8_t s2,
+ const int16x8_t s3, const int16x8_t s4, const int16x8_t s5,
+ const int16x8_t s6, const int16x8_t s7, const int16x8_t filter,
+ const int16x8_t horiz_const, const int16x8_t shift_round_0) {
+ const int16x4_t filter_lo = vget_low_s16(filter);
+ const int16x4_t filter_hi = vget_high_s16(filter);
int16x8_t sum;
- int16x8_t res;
sum = horiz_const;
- sum = vmlaq_n_s16(sum, s0, filter[0]);
- sum = vmlaq_n_s16(sum, s1, filter[1]);
- sum = vmlaq_n_s16(sum, s2, filter[2]);
- sum = vmlaq_n_s16(sum, s3, filter[3]);
- sum = vmlaq_n_s16(sum, s4, filter[4]);
- sum = vmlaq_n_s16(sum, s5, filter[5]);
- sum = vmlaq_n_s16(sum, s6, filter[6]);
- sum = vmlaq_n_s16(sum, s7, filter[7]);
+ sum = vmlaq_lane_s16(sum, s0, filter_lo, 0);
+ sum = vmlaq_lane_s16(sum, s1, filter_lo, 1);
+ sum = vmlaq_lane_s16(sum, s2, filter_lo, 2);
+ sum = vmlaq_lane_s16(sum, s3, filter_lo, 3);
+ sum = vmlaq_lane_s16(sum, s4, filter_hi, 0);
+ sum = vmlaq_lane_s16(sum, s5, filter_hi, 1);
+ sum = vmlaq_lane_s16(sum, s6, filter_hi, 2);
+ sum = vmlaq_lane_s16(sum, s7, filter_hi, 3);
- res = vqrshlq_s16(sum, shift_round_0);
+ sum = vqrshlq_s16(sum, shift_round_0);
- return res;
+ return sum;
}
-static INLINE int16x4_t
-convolve8_4x4_s16(const int16x4_t s0, const int16x4_t s1, const int16x4_t s2,
- const int16x4_t s3, const int16x4_t s4, const int16x4_t s5,
- const int16x4_t s6, const int16x4_t s7, const int16_t *filter,
- const int16x4_t horiz_const, const int16x4_t shift_round_0) {
+#if defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+
+DECLARE_ALIGNED(16, static const uint8_t, dot_prod_permute_tbl[48]) = {
+ 0, 1, 2, 3, 1, 2, 3, 4, 2, 3, 4, 5, 3, 4, 5, 6,
+ 4, 5, 6, 7, 5, 6, 7, 8, 6, 7, 8, 9, 7, 8, 9, 10,
+ 8, 9, 10, 11, 9, 10, 11, 12, 10, 11, 12, 13, 11, 12, 13, 14
+};
+
+static INLINE int32x4_t convolve8_4_dot_s16(uint8x16_t samples,
+ const int8x8_t filters,
+ const int32x4_t correction,
+ const uint8x16_t range_limit,
+ const uint8x16x2_t permute_tbl) {
+ int8x16_t clamped_samples, permuted_samples[2];
+ int32x4_t sum;
+
+ /* Clamp sample range to [-128, 127] for 8-bit signed dot product. */
+ clamped_samples = vreinterpretq_s8_u8(vsubq_u8(samples, range_limit));
+
+ /* Permute samples ready for dot product. */
+ /* { 0, 1, 2, 3, 1, 2, 3, 4, 2, 3, 4, 5, 3, 4, 5, 6 } */
+ permuted_samples[0] = vqtbl1q_s8(clamped_samples, permute_tbl.val[0]);
+ /* { 4, 5, 6, 7, 5, 6, 7, 8, 6, 7, 8, 9, 7, 8, 9, 10 } */
+ permuted_samples[1] = vqtbl1q_s8(clamped_samples, permute_tbl.val[1]);
+
+ /* Accumulate dot product into 'correction' to account for range clamp. */
+ sum = vdotq_lane_s32(correction, permuted_samples[0], filters, 0);
+ sum = vdotq_lane_s32(sum, permuted_samples[1], filters, 1);
+
+ /* Narrowing and packing is performed by the caller. */
+ return sum;
+}
+
+static INLINE int16x8_t convolve8_8_dot_s16(uint8x16_t samples,
+ const int8x8_t filters,
+ const int32x4_t correction,
+ const uint8x16_t range_limit,
+ const uint8x16x3_t permute_tbl,
+ const int16x8_t shift_round_0) {
+ int8x16_t clamped_samples, permuted_samples[3];
+ int32x4_t sum0, sum1;
+ int16x8_t sum;
+
+ /* Clamp sample range to [-128, 127] for 8-bit signed dot product. */
+ clamped_samples = vreinterpretq_s8_u8(vsubq_u8(samples, range_limit));
+
+ /* Permute samples ready for dot product. */
+ /* { 0, 1, 2, 3, 1, 2, 3, 4, 2, 3, 4, 5, 3, 4, 5, 6 } */
+ permuted_samples[0] = vqtbl1q_s8(clamped_samples, permute_tbl.val[0]);
+ /* { 4, 5, 6, 7, 5, 6, 7, 8, 6, 7, 8, 9, 7, 8, 9, 10 } */
+ permuted_samples[1] = vqtbl1q_s8(clamped_samples, permute_tbl.val[1]);
+ /* { 8, 9, 10, 11, 9, 10, 11, 12, 10, 11, 12, 13, 11, 12, 13, 14 } */
+ permuted_samples[2] = vqtbl1q_s8(clamped_samples, permute_tbl.val[2]);
+
+ /* Accumulate dot product into 'correction' to account for range clamp. */
+ /* First 4 output values. */
+ sum0 = vdotq_lane_s32(correction, permuted_samples[0], filters, 0);
+ sum0 = vdotq_lane_s32(sum0, permuted_samples[1], filters, 1);
+ /* Second 4 output values. */
+ sum1 = vdotq_lane_s32(correction, permuted_samples[1], filters, 0);
+ sum1 = vdotq_lane_s32(sum1, permuted_samples[2], filters, 1);
+
+ /* Narrow and re-pack. */
+ sum = vcombine_s16(vmovn_s32(sum0), vmovn_s32(sum1));
+ return vqrshlq_s16(sum, shift_round_0);
+}
+
+#endif // defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+
+static INLINE int16x4_t convolve8_4x4_s16(
+ const int16x4_t s0, const int16x4_t s1, const int16x4_t s2,
+ const int16x4_t s3, const int16x4_t s4, const int16x4_t s5,
+ const int16x4_t s6, const int16x4_t s7, const int16x8_t filter,
+ const int16x4_t horiz_const, const int16x4_t shift_round_0) {
+ const int16x4_t filter_lo = vget_low_s16(filter);
+ const int16x4_t filter_hi = vget_high_s16(filter);
int16x4_t sum;
+
sum = horiz_const;
- sum = vmla_n_s16(sum, s0, filter[0]);
- sum = vmla_n_s16(sum, s1, filter[1]);
- sum = vmla_n_s16(sum, s2, filter[2]);
- sum = vmla_n_s16(sum, s3, filter[3]);
- sum = vmla_n_s16(sum, s4, filter[4]);
- sum = vmla_n_s16(sum, s5, filter[5]);
- sum = vmla_n_s16(sum, s6, filter[6]);
- sum = vmla_n_s16(sum, s7, filter[7]);
+ sum = vmla_lane_s16(sum, s0, filter_lo, 0);
+ sum = vmla_lane_s16(sum, s1, filter_lo, 1);
+ sum = vmla_lane_s16(sum, s2, filter_lo, 2);
+ sum = vmla_lane_s16(sum, s3, filter_lo, 3);
+ sum = vmla_lane_s16(sum, s4, filter_hi, 0);
+ sum = vmla_lane_s16(sum, s5, filter_hi, 1);
+ sum = vmla_lane_s16(sum, s6, filter_hi, 2);
+ sum = vmla_lane_s16(sum, s7, filter_hi, 3);
sum = vqrshl_s16(sum, shift_round_0);
@@ -265,27 +326,25 @@
static INLINE uint16x4_t convolve8_4x4_s32(
const int16x4_t s0, const int16x4_t s1, const int16x4_t s2,
const int16x4_t s3, const int16x4_t s4, const int16x4_t s5,
- const int16x4_t s6, const int16x4_t s7, const int16_t *y_filter,
+ const int16x4_t s6, const int16x4_t s7, const int16x8_t y_filter,
const int32x4_t round_shift_vec, const int32x4_t offset_const) {
- int32x4_t sum0;
- uint16x4_t res;
- const int32x4_t zero = vdupq_n_s32(0);
+ const int16x4_t y_filter_lo = vget_low_s16(y_filter);
+ const int16x4_t y_filter_hi = vget_high_s16(y_filter);
+ int32x4_t sum;
- sum0 = vmull_n_s16(s0, y_filter[0]);
- sum0 = vmlal_n_s16(sum0, s1, y_filter[1]);
- sum0 = vmlal_n_s16(sum0, s2, y_filter[2]);
- sum0 = vmlal_n_s16(sum0, s3, y_filter[3]);
- sum0 = vmlal_n_s16(sum0, s4, y_filter[4]);
- sum0 = vmlal_n_s16(sum0, s5, y_filter[5]);
- sum0 = vmlal_n_s16(sum0, s6, y_filter[6]);
- sum0 = vmlal_n_s16(sum0, s7, y_filter[7]);
+ sum = vmull_lane_s16(s0, y_filter_lo, 0);
+ sum = vmlal_lane_s16(sum, s1, y_filter_lo, 1);
+ sum = vmlal_lane_s16(sum, s2, y_filter_lo, 2);
+ sum = vmlal_lane_s16(sum, s3, y_filter_lo, 3);
+ sum = vmlal_lane_s16(sum, s4, y_filter_hi, 0);
+ sum = vmlal_lane_s16(sum, s5, y_filter_hi, 1);
+ sum = vmlal_lane_s16(sum, s6, y_filter_hi, 2);
+ sum = vmlal_lane_s16(sum, s7, y_filter_hi, 3);
- sum0 = vaddq_s32(sum0, offset_const);
- sum0 = vqrshlq_s32(sum0, round_shift_vec);
- sum0 = vmaxq_s32(sum0, zero);
- res = vmovn_u32(vreinterpretq_u32_s32(sum0));
+ sum = vaddq_s32(sum, offset_const);
+ sum = vqrshlq_s32(sum, round_shift_vec);
- return res;
+ return vqmovun_s32(sum);
}
#endif // AOM_AV1_COMMON_ARM_CONVOLVE_NEON_H_
diff --git a/av1/common/arm/jnt_convolve_neon.c b/av1/common/arm/jnt_convolve_neon.c
index e0b76a8..dd35fdf 100644
--- a/av1/common/arm/jnt_convolve_neon.c
+++ b/av1/common/arm/jnt_convolve_neon.c
@@ -45,7 +45,7 @@
dst0 = vqrshlq_s32(dst0, round_bits_vec);
- tmp0 = vqmovn_s32(dst0);
+ tmp0 = vmovn_s32(dst0);
tmp4 = vcombine_s16(tmp0, tmp0);
*t0 = vqmovun_s16(tmp4);
@@ -57,7 +57,7 @@
tmp0 = vqrshl_s16(tmp0, round_bits_vec);
- tmp4 = vcombine_s16(tmp0, tmp0);
+ tmp4 = vcombine_s16(tmp0, vdup_n_s16(0));
*t0 = vqmovun_s16(tmp4);
}
@@ -67,7 +67,6 @@
uint16x8_t res0, uint16x8_t d0, const uint16_t fwd_offset,
const uint16_t bck_offset, const int16x4_t sub_const,
const int16_t round_bits, const int use_dist_wtd_comp_avg, uint8x8_t *t0) {
- int16x4_t tmp0, tmp2;
int16x8_t f0;
uint32x4_t sum0, sum2;
int32x4_t dst0, dst2;
@@ -92,10 +91,7 @@
dst0 = vqrshlq_s32(dst0, round_bits_vec);
dst2 = vqrshlq_s32(dst2, round_bits_vec);
- tmp0 = vqmovn_s32(dst0);
- tmp2 = vqmovn_s32(dst2);
-
- f0 = vcombine_s16(tmp0, tmp2);
+ f0 = vcombine_s16(vmovn_s32(dst0), vmovn_s32(dst2));
*t0 = vqmovun_s16(f0);
@@ -126,7 +122,6 @@
int32x4_t dst0, dst1, dst2, dst3;
int16x8_t tmp4, tmp5;
- const int16x8_t zero = vdupq_n_s16(0);
if (use_dist_wtd_comp_avg) {
const int32x4_t round_bits_vec = vdupq_n_s32((int32_t)(-round_bits));
@@ -156,17 +151,11 @@
dst2 = vqrshlq_s32(dst2, round_bits_vec);
dst3 = vqrshlq_s32(dst3, round_bits_vec);
- tmp0 = vqmovn_s32(dst0);
- tmp1 = vqmovn_s32(dst1);
- tmp2 = vqmovn_s32(dst2);
- tmp3 = vqmovn_s32(dst3);
- tmp4 = vcombine_s16(tmp0, tmp1);
- tmp5 = vcombine_s16(tmp2, tmp3);
- tmp4 = vmaxq_s16(tmp4, zero);
- tmp5 = vmaxq_s16(tmp5, zero);
+ tmp4 = vcombine_s16(vmovn_s32(dst0), vmovn_s32(dst1));
+ tmp5 = vcombine_s16(vmovn_s32(dst2), vmovn_s32(dst3));
- *t0 = vqmovn_u16(vreinterpretq_u16_s16(tmp4));
- *t1 = vqmovn_u16(vreinterpretq_u16_s16(tmp5));
+ *t0 = vqmovun_s16(tmp4);
+ *t1 = vqmovun_s16(tmp5);
} else {
const int16x4_t round_bits_vec = vdup_n_s16(-round_bits);
tmp_u0 = vhadd_u16(res0, d0);
@@ -186,11 +175,9 @@
tmp4 = vcombine_s16(tmp0, tmp1);
tmp5 = vcombine_s16(tmp2, tmp3);
- tmp4 = vmaxq_s16(tmp4, zero);
- tmp5 = vmaxq_s16(tmp5, zero);
- *t0 = vqmovn_u16(vreinterpretq_u16_s16(tmp4));
- *t1 = vqmovn_u16(vreinterpretq_u16_s16(tmp5));
+ *t0 = vqmovun_s16(tmp4);
+ *t1 = vqmovun_s16(tmp5);
}
}
@@ -201,14 +188,12 @@
const int16x4_t sub_const, const int16_t round_bits,
const int use_dist_wtd_comp_avg, uint8x8_t *t0, uint8x8_t *t1,
uint8x8_t *t2, uint8x8_t *t3) {
- int16x4_t tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7;
int16x8_t f0, f1, f2, f3;
uint32x4_t sum0, sum1, sum2, sum3;
uint32x4_t sum4, sum5, sum6, sum7;
int32x4_t dst0, dst1, dst2, dst3;
int32x4_t dst4, dst5, dst6, dst7;
uint16x8_t tmp_u0, tmp_u1, tmp_u2, tmp_u3;
- const int16x8_t zero = vdupq_n_s16(0);
if (use_dist_wtd_comp_avg) {
const int32x4_t sub_const_vec = vmovl_s16(sub_const);
@@ -260,29 +245,15 @@
dst6 = vqrshlq_s32(dst6, round_bits_vec);
dst7 = vqrshlq_s32(dst7, round_bits_vec);
- tmp0 = vqmovn_s32(dst0);
- tmp1 = vqmovn_s32(dst1);
- tmp2 = vqmovn_s32(dst2);
- tmp3 = vqmovn_s32(dst3);
- tmp4 = vqmovn_s32(dst4);
- tmp5 = vqmovn_s32(dst5);
- tmp6 = vqmovn_s32(dst6);
- tmp7 = vqmovn_s32(dst7);
+ f0 = vcombine_s16(vmovn_s32(dst0), vmovn_s32(dst2));
+ f1 = vcombine_s16(vmovn_s32(dst1), vmovn_s32(dst3));
+ f2 = vcombine_s16(vmovn_s32(dst4), vmovn_s32(dst6));
+ f3 = vcombine_s16(vmovn_s32(dst5), vmovn_s32(dst7));
- f0 = vcombine_s16(tmp0, tmp2);
- f1 = vcombine_s16(tmp1, tmp3);
- f2 = vcombine_s16(tmp4, tmp6);
- f3 = vcombine_s16(tmp5, tmp7);
-
- f0 = vmaxq_s16(f0, zero);
- f1 = vmaxq_s16(f1, zero);
- f2 = vmaxq_s16(f2, zero);
- f3 = vmaxq_s16(f3, zero);
-
- *t0 = vqmovn_u16(vreinterpretq_u16_s16(f0));
- *t1 = vqmovn_u16(vreinterpretq_u16_s16(f1));
- *t2 = vqmovn_u16(vreinterpretq_u16_s16(f2));
- *t3 = vqmovn_u16(vreinterpretq_u16_s16(f3));
+ *t0 = vqmovun_s16(f0);
+ *t1 = vqmovun_s16(f1);
+ *t2 = vqmovun_s16(f2);
+ *t3 = vqmovun_s16(f3);
} else {
const int16x8_t sub_const_vec = vcombine_s16(sub_const, sub_const);
@@ -303,21 +274,118 @@
f2 = vqrshlq_s16(f2, round_bits_vec);
f3 = vqrshlq_s16(f3, round_bits_vec);
- f0 = vmaxq_s16(f0, zero);
- f1 = vmaxq_s16(f1, zero);
- f2 = vmaxq_s16(f2, zero);
- f3 = vmaxq_s16(f3, zero);
-
- *t0 = vqmovn_u16(vreinterpretq_u16_s16(f0));
- *t1 = vqmovn_u16(vreinterpretq_u16_s16(f1));
- *t2 = vqmovn_u16(vreinterpretq_u16_s16(f2));
- *t3 = vqmovn_u16(vreinterpretq_u16_s16(f3));
+ *t0 = vqmovun_s16(f0);
+ *t1 = vqmovun_s16(f1);
+ *t2 = vqmovun_s16(f2);
+ *t3 = vqmovun_s16(f3);
}
}
+#if defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+
static INLINE void dist_wtd_convolve_2d_horiz_neon(
const uint8_t *src, int src_stride, int16_t *im_block, const int im_stride,
- int16_t *x_filter_tmp, const int im_h, int w, const int round_0) {
+ const int16x8_t x_filter_s16, const int im_h, int w, const int round_0) {
+ const int bd = 8;
+ int16_t *dst_ptr = im_block;
+ int dst_stride = im_stride;
+ int width = w;
+ int height = im_h;
+
+ const int8x8_t x_filter = vmovn_s16(x_filter_s16);
+ const int32_t horiz_const = (1 << (bd + FILTER_BITS - 2));
+ // Dot product constants.
+ const int16x8_t correct_tmp = vshlq_n_s16(x_filter_s16, 7);
+ const int32x4_t correction =
+ vdupq_n_s32(vaddlvq_s16(correct_tmp) + horiz_const);
+ const uint8x16_t range_limit = vdupq_n_u8(128);
+
+ if (w == 4) {
+ const uint8x16x2_t permute_tbl = vld1q_u8_x2(dot_prod_permute_tbl);
+ const int16x4_t shift_round_0 = vdup_n_s16(-(round_0));
+ uint8x16_t s0, s1, s2, s3;
+ int32x4_t t0, t1, t2, t3;
+ int16x4_t d0, d1, d2, d3;
+
+ do {
+ s0 = vld1q_u8(src + 0 * src_stride);
+ s1 = vld1q_u8(src + 1 * src_stride);
+ s2 = vld1q_u8(src + 2 * src_stride);
+ s3 = vld1q_u8(src + 3 * src_stride);
+
+ t0 = convolve8_4_dot_s16(s0, x_filter, correction, range_limit,
+ permute_tbl);
+ t1 = convolve8_4_dot_s16(s1, x_filter, correction, range_limit,
+ permute_tbl);
+ t2 = convolve8_4_dot_s16(s2, x_filter, correction, range_limit,
+ permute_tbl);
+ t3 = convolve8_4_dot_s16(s3, x_filter, correction, range_limit,
+ permute_tbl);
+
+ d0 = vqrshl_s16(vmovn_s32(t0), shift_round_0);
+ d1 = vqrshl_s16(vmovn_s32(t1), shift_round_0);
+ d2 = vqrshl_s16(vmovn_s32(t2), shift_round_0);
+ d3 = vqrshl_s16(vmovn_s32(t3), shift_round_0);
+
+ vst1_s16((dst_ptr + 0 * dst_stride), d0);
+ vst1_s16((dst_ptr + 1 * dst_stride), d1);
+ vst1_s16((dst_ptr + 2 * dst_stride), d2);
+ vst1_s16((dst_ptr + 3 * dst_stride), d3);
+
+ src += 4 * src_stride;
+ dst_ptr += 4 * dst_stride;
+ height -= 4;
+ } while (height > 0);
+ } else {
+ const uint8x16x3_t permute_tbl = vld1q_u8_x3(dot_prod_permute_tbl);
+ const int16x8_t shift_round_0 = vdupq_n_s16(-(round_0));
+ const uint8_t *s;
+ int16_t *d;
+ uint8x16_t s0, s1, s2, s3;
+ int16x8_t d0, d1, d2, d3;
+
+ do {
+ width = w;
+ s = src;
+ d = dst_ptr;
+
+ do {
+ s0 = vld1q_u8(s + 0 * src_stride);
+ s1 = vld1q_u8(s + 1 * src_stride);
+ s2 = vld1q_u8(s + 2 * src_stride);
+ s3 = vld1q_u8(s + 3 * src_stride);
+
+ d0 = convolve8_8_dot_s16(s0, x_filter, correction, range_limit,
+ permute_tbl, shift_round_0);
+ d1 = convolve8_8_dot_s16(s1, x_filter, correction, range_limit,
+ permute_tbl, shift_round_0);
+ d2 = convolve8_8_dot_s16(s2, x_filter, correction, range_limit,
+ permute_tbl, shift_round_0);
+ d3 = convolve8_8_dot_s16(s3, x_filter, correction, range_limit,
+ permute_tbl, shift_round_0);
+
+ vst1q_s16(d + 0 * dst_stride, d0);
+ vst1q_s16(d + 1 * dst_stride, d1);
+ vst1q_s16(d + 2 * dst_stride, d2);
+ vst1q_s16(d + 3 * dst_stride, d3);
+
+ s += 8;
+ d += 8;
+ width -= 8;
+ } while (width > 0);
+
+ src += 4 * src_stride;
+ dst_ptr += 4 * dst_stride;
+ height -= 4;
+ } while (height > 0);
+ }
+}
+
+#else // !(defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD))
+
+static INLINE void dist_wtd_convolve_2d_horiz_neon(
+ const uint8_t *src, int src_stride, int16_t *im_block, const int im_stride,
+ const int16x8_t x_filter, const int im_h, int w, const int round_0) {
const int bd = 8;
const uint8_t *s;
int16_t *dst_ptr;
@@ -380,13 +448,13 @@
s9 = vget_low_s16(tt2);
s10 = vget_low_s16(tt3);
- d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter_tmp,
+ d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter,
horiz_const, shift_round_0);
- d1 = convolve8_4x4_s16(s1, s2, s3, s4, s5, s6, s7, s8, x_filter_tmp,
+ d1 = convolve8_4x4_s16(s1, s2, s3, s4, s5, s6, s7, s8, x_filter,
horiz_const, shift_round_0);
- d2 = convolve8_4x4_s16(s2, s3, s4, s5, s6, s7, s8, s9, x_filter_tmp,
+ d2 = convolve8_4x4_s16(s2, s3, s4, s5, s6, s7, s8, s9, x_filter,
horiz_const, shift_round_0);
- d3 = convolve8_4x4_s16(s3, s4, s5, s6, s7, s8, s9, s10, x_filter_tmp,
+ d3 = convolve8_4x4_s16(s3, s4, s5, s6, s7, s8, s9, s10, x_filter,
horiz_const, shift_round_0);
transpose_s16_4x4d(&d0, &d1, &d2, &d3);
@@ -418,7 +486,7 @@
s6 = vext_s16(s4, s7, 2); // a6 a7 a8 a9
s7 = vext_s16(s4, s7, 3); // a7 a8 a9 a10
- d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter_tmp,
+ d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter,
horiz_const, shift_round_0);
vst1_s16(dst_ptr, d0);
@@ -483,22 +551,22 @@
s13 = vreinterpretq_s16_u16(vmovl_u8(t6));
s14 = vreinterpretq_s16_u16(vmovl_u8(t7));
- res0 = convolve8_8x8_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter_tmp,
+ res0 = convolve8_8x8_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter,
horiz_const, shift_round_0);
- res1 = convolve8_8x8_s16(s1, s2, s3, s4, s5, s6, s7, s8, x_filter_tmp,
+ res1 = convolve8_8x8_s16(s1, s2, s3, s4, s5, s6, s7, s8, x_filter,
horiz_const, shift_round_0);
- res2 = convolve8_8x8_s16(s2, s3, s4, s5, s6, s7, s8, s9, x_filter_tmp,
+ res2 = convolve8_8x8_s16(s2, s3, s4, s5, s6, s7, s8, s9, x_filter,
horiz_const, shift_round_0);
- res3 = convolve8_8x8_s16(s3, s4, s5, s6, s7, s8, s9, s10, x_filter_tmp,
+ res3 = convolve8_8x8_s16(s3, s4, s5, s6, s7, s8, s9, s10, x_filter,
horiz_const, shift_round_0);
- res4 = convolve8_8x8_s16(s4, s5, s6, s7, s8, s9, s10, s11, x_filter_tmp,
+ res4 = convolve8_8x8_s16(s4, s5, s6, s7, s8, s9, s10, s11, x_filter,
horiz_const, shift_round_0);
- res5 = convolve8_8x8_s16(s5, s6, s7, s8, s9, s10, s11, s12,
- x_filter_tmp, horiz_const, shift_round_0);
- res6 = convolve8_8x8_s16(s6, s7, s8, s9, s10, s11, s12, s13,
- x_filter_tmp, horiz_const, shift_round_0);
- res7 = convolve8_8x8_s16(s7, s8, s9, s10, s11, s12, s13, s14,
- x_filter_tmp, horiz_const, shift_round_0);
+ res5 = convolve8_8x8_s16(s5, s6, s7, s8, s9, s10, s11, s12, x_filter,
+ horiz_const, shift_round_0);
+ res6 = convolve8_8x8_s16(s6, s7, s8, s9, s10, s11, s12, s13, x_filter,
+ horiz_const, shift_round_0);
+ res7 = convolve8_8x8_s16(s7, s8, s9, s10, s11, s12, s13, s14, x_filter,
+ horiz_const, shift_round_0);
transpose_s16_8x8(&res0, &res1, &res2, &res3, &res4, &res5, &res6,
&res7);
@@ -543,8 +611,8 @@
s6 = vextq_s16(temp_0, s7, 6); // a6 a7 a8 a9 a10 a11 a12 a13
s7 = vextq_s16(temp_0, s7, 7); // a7 a8 a9 a10 a11 a12 a13 a14
- res0 = convolve8_8x8_s16(temp_0, s1, s2, s3, s4, s5, s6, s7,
- x_filter_tmp, horiz_const, shift_round_0);
+ res0 = convolve8_8x8_s16(temp_0, s1, s2, s3, s4, s5, s6, s7, x_filter,
+ horiz_const, shift_round_0);
vst1q_s16(d_tmp, res0);
s += 8;
@@ -559,9 +627,11 @@
}
}
+#endif // defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+
static INLINE void dist_wtd_convolve_2d_vert_neon(
int16_t *im_block, const int im_stride, uint8_t *dst8, int dst8_stride,
- ConvolveParams *conv_params, const int16_t *y_filter, int h, int w) {
+ ConvolveParams *conv_params, const int16x8_t y_filter, int h, int w) {
uint8_t *dst_u8_ptr, *d_u8;
CONV_BUF_TYPE *dst_ptr, *dst;
int16_t *src_ptr, *s;
@@ -731,21 +801,18 @@
const int horiz_offset = filter_params_x->taps / 2 - 1;
const int round_0 = conv_params->round_0 - 1;
const uint8_t *src_ptr = src - vert_offset * src_stride - horiz_offset;
- const int16_t *x_filter = av1_get_interp_filter_subpel_kernel(
+ const int16_t *x_filter_ptr = av1_get_interp_filter_subpel_kernel(
filter_params_x, subpel_x_qn & SUBPEL_MASK);
- const int16_t *y_filter = av1_get_interp_filter_subpel_kernel(
+ const int16_t *y_filter_ptr = av1_get_interp_filter_subpel_kernel(
filter_params_y, subpel_y_qn & SUBPEL_MASK);
- int16_t x_filter_tmp[8];
- int16x8_t filter_x_coef = vld1q_s16(x_filter);
-
- // filter coeffs are even, so downshifting by 1 to reduce intermediate
- // precision requirements.
- filter_x_coef = vshrq_n_s16(filter_x_coef, 1);
- vst1q_s16(&x_filter_tmp[0], filter_x_coef);
+ // Filter values are even, so downshift by 1 to reduce intermediate precision
+ // requirements.
+ const int16x8_t x_filter = vshrq_n_s16(vld1q_s16(x_filter_ptr), 1);
+ const int16x8_t y_filter = vld1q_s16(y_filter_ptr);
dist_wtd_convolve_2d_horiz_neon(src_ptr, src_stride, im_block, im_stride,
- x_filter_tmp, im_h, w, round_0);
+ x_filter, im_h, w, round_0);
dist_wtd_convolve_2d_vert_neon(im_block, im_stride, dst8, dst8_stride,
conv_params, y_filter, h, w);
@@ -869,6 +936,198 @@
}
}
+#if defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+
+void av1_dist_wtd_convolve_x_neon(const uint8_t *src, int src_stride,
+ uint8_t *dst8, int dst8_stride, int w, int h,
+ const InterpFilterParams *filter_params_x,
+ const int subpel_x_qn,
+ ConvolveParams *conv_params) {
+ assert(!(w % 4));
+ assert(!(h % 4));
+
+ const int horiz_offset = filter_params_x->taps / 2 - 1;
+ const int bits = FILTER_BITS - conv_params->round_1;
+ const int bd = 8;
+ const int offset_bits = bd + 2 * FILTER_BITS - conv_params->round_0;
+ const int round_offset = (1 << (offset_bits - conv_params->round_1)) +
+ (1 << (offset_bits - conv_params->round_1 - 1));
+ const int round_bits =
+ 2 * FILTER_BITS - conv_params->round_0 - conv_params->round_1;
+ const uint16_t fwd_offset = conv_params->fwd_offset;
+ const uint16_t bck_offset = conv_params->bck_offset;
+ const int use_dist_wtd_comp_avg = conv_params->use_dist_wtd_comp_avg;
+ const int16x4_t round_offset64 = vdup_n_s16(round_offset);
+ const int16x8_t round_offset128 = vdupq_n_s16(round_offset);
+ const int16x8_t shift_round_0 = vdupq_n_s16(-conv_params->round_0 + 1);
+ const int16x8_t horiz_const = vdupq_n_s16(bits);
+
+ // Horizontal filter.
+ const int16_t *x_filter_ptr = av1_get_interp_filter_subpel_kernel(
+ filter_params_x, subpel_x_qn & SUBPEL_MASK);
+ // Filter values are even, so downshift by 1 to reduce intermediate precision
+ // requirements.
+ const int8x8_t x_filter = vshrn_n_s16(vld1q_s16(x_filter_ptr), 1);
+ // Dot-product constants.
+ const uint8x16_t range_limit = vdupq_n_u8(128);
+ const int32_t correction_s32 = vaddlvq_s16(vshll_n_s8(x_filter, 7));
+ const int32x4_t correction = vdupq_n_s32(correction_s32);
+
+ const uint8_t *src_ptr = src - horiz_offset;
+ CONV_BUF_TYPE *dst = conv_params->dst;
+ CONV_BUF_TYPE *dst_ptr = dst;
+ uint8_t *dst_u8_ptr = dst8;
+ int dst_stride = conv_params->dst_stride;
+ int width = w;
+ int height = h;
+
+ if (w == 4) {
+ const uint8x16x2_t permute_tbl = vld1q_u8_x2(dot_prod_permute_tbl);
+
+ do {
+ uint8x16_t s0, s1, s2, s3;
+ int32x4_t d0, d1, d2, d3;
+ int16x8_t d01, d23;
+ uint16x4_t dd0, dd1, dd2, dd3;
+ uint8x8_t d01_u8, d23_u8;
+
+ s0 = vld1q_u8(src_ptr + 0 * src_stride);
+ s1 = vld1q_u8(src_ptr + 1 * src_stride);
+ s2 = vld1q_u8(src_ptr + 2 * src_stride);
+ s3 = vld1q_u8(src_ptr + 3 * src_stride);
+
+ d0 = convolve8_4_dot_s16(s0, x_filter, correction, range_limit,
+ permute_tbl);
+ d1 = convolve8_4_dot_s16(s1, x_filter, correction, range_limit,
+ permute_tbl);
+ d2 = convolve8_4_dot_s16(s2, x_filter, correction, range_limit,
+ permute_tbl);
+ d3 = convolve8_4_dot_s16(s3, x_filter, correction, range_limit,
+ permute_tbl);
+
+ d01 = vcombine_s16(vmovn_s32(d0), vmovn_s32(d1));
+ d23 = vcombine_s16(vmovn_s32(d2), vmovn_s32(d3));
+
+ d01 = vqrshlq_s16(d01, shift_round_0);
+ d23 = vqrshlq_s16(d23, shift_round_0);
+
+ d01 = vrshlq_s16(d01, horiz_const);
+ d23 = vrshlq_s16(d23, horiz_const);
+
+ d01 = vaddq_s16(d01, round_offset128);
+ d23 = vaddq_s16(d23, round_offset128);
+
+ if (conv_params->do_average) {
+ dd0 = vld1_u16(dst_ptr);
+ dst_ptr += dst_stride;
+ dd1 = vld1_u16(dst_ptr);
+ dst_ptr += dst_stride;
+ dd2 = vld1_u16(dst_ptr);
+ dst_ptr += dst_stride;
+ dd3 = vld1_u16(dst_ptr);
+ dst_ptr += dst_stride;
+
+ compute_avg_4x4(dd0, dd1, dd2, dd3,
+ vreinterpret_u16_s16(vget_low_s16(d01)),
+ vreinterpret_u16_s16(vget_high_s16(d01)),
+ vreinterpret_u16_s16(vget_low_s16(d23)),
+ vreinterpret_u16_s16(vget_high_s16(d23)), fwd_offset,
+ bck_offset, round_offset64, round_bits,
+ use_dist_wtd_comp_avg, &d01_u8, &d23_u8);
+
+ vst1_lane_u32((uint32_t *)dst_u8_ptr, vreinterpret_u32_u8(d01_u8), 0);
+ dst_u8_ptr += dst8_stride;
+ vst1_lane_u32((uint32_t *)dst_u8_ptr, vreinterpret_u32_u8(d01_u8), 1);
+ dst_u8_ptr += dst8_stride;
+ vst1_lane_u32((uint32_t *)dst_u8_ptr, vreinterpret_u32_u8(d23_u8), 0);
+ dst_u8_ptr += dst8_stride;
+ vst1_lane_u32((uint32_t *)dst_u8_ptr, vreinterpret_u32_u8(d23_u8), 1);
+ dst_u8_ptr += dst8_stride;
+ } else {
+ vst1q_lane_u64((uint64_t *)dst_ptr, vreinterpretq_u64_s16(d01), 0);
+ dst_ptr += dst_stride;
+ vst1q_lane_u64((uint64_t *)dst_ptr, vreinterpretq_u64_s16(d01), 1);
+ dst_ptr += dst_stride;
+ vst1q_lane_u64((uint64_t *)dst_ptr, vreinterpretq_u64_s16(d23), 0);
+ dst_ptr += dst_stride;
+ vst1q_lane_u64((uint64_t *)dst_ptr, vreinterpretq_u64_s16(d23), 1);
+ dst_ptr += dst_stride;
+ }
+
+ src_ptr += 4 * src_stride;
+ height -= 4;
+ } while (height > 0);
+ } else {
+ const uint8x16x3_t permute_tbl = vld1q_u8_x3(dot_prod_permute_tbl);
+
+ do {
+ const uint8_t *s = src_ptr;
+ CONV_BUF_TYPE *d = dst_ptr;
+ uint8_t *d_u8 = dst_u8_ptr;
+ width = w;
+
+ do {
+ uint8x16_t s0, s1, s2, s3;
+ int16x8_t d0, d1, d2, d3;
+ uint16x8_t dd0, dd1, dd2, dd3;
+ uint8x8_t d0_u8, d1_u8, d2_u8, d3_u8;
+
+ s0 = vld1q_u8(s + 0 * src_stride);
+ s1 = vld1q_u8(s + 1 * src_stride);
+ s2 = vld1q_u8(s + 2 * src_stride);
+ s3 = vld1q_u8(s + 3 * src_stride);
+
+ d0 = convolve8_8_dot_s16(s0, x_filter, correction, range_limit,
+ permute_tbl, shift_round_0);
+ d1 = convolve8_8_dot_s16(s1, x_filter, correction, range_limit,
+ permute_tbl, shift_round_0);
+ d2 = convolve8_8_dot_s16(s2, x_filter, correction, range_limit,
+ permute_tbl, shift_round_0);
+ d3 = convolve8_8_dot_s16(s3, x_filter, correction, range_limit,
+ permute_tbl, shift_round_0);
+
+ d0 = vrshlq_s16(d0, horiz_const);
+ d1 = vrshlq_s16(d1, horiz_const);
+ d2 = vrshlq_s16(d2, horiz_const);
+ d3 = vrshlq_s16(d3, horiz_const);
+
+ d0 = vaddq_s16(d0, round_offset128);
+ d1 = vaddq_s16(d1, round_offset128);
+ d2 = vaddq_s16(d2, round_offset128);
+ d3 = vaddq_s16(d3, round_offset128);
+
+ if (conv_params->do_average) {
+ load_u16_8x4(d, dst_stride, &dd0, &dd1, &dd2, &dd3);
+
+ compute_avg_8x4(dd0, dd1, dd2, dd3, vreinterpretq_u16_s16(d0),
+ vreinterpretq_u16_s16(d1), vreinterpretq_u16_s16(d2),
+ vreinterpretq_u16_s16(d3), fwd_offset, bck_offset,
+ round_offset64, round_bits, use_dist_wtd_comp_avg,
+ &d0_u8, &d1_u8, &d2_u8, &d3_u8);
+
+ store_u8_8x4(d_u8, dst8_stride, d0_u8, d1_u8, d2_u8, d3_u8);
+ } else {
+ store_u16_8x4(d, dst_stride, vreinterpretq_u16_s16(d0),
+ vreinterpretq_u16_s16(d1), vreinterpretq_u16_s16(d2),
+ vreinterpretq_u16_s16(d3));
+ }
+
+ s += 8;
+ d += 8;
+ d_u8 += 8;
+ width -= 8;
+ } while (width > 0);
+
+ src_ptr += 4 * src_stride;
+ dst_ptr += 4 * dst_stride;
+ dst_u8_ptr += 4 * dst8_stride;
+ height -= 4;
+ } while (height > 0);
+ }
+}
+
+#else // !(defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD))
+
void av1_dist_wtd_convolve_x_neon(const uint8_t *src, int src_stride,
uint8_t *dst8, int dst8_stride, int w, int h,
const InterpFilterParams *filter_params_x,
@@ -892,18 +1151,14 @@
const int use_dist_wtd_comp_avg = conv_params->use_dist_wtd_comp_avg;
// horizontal filter
- const int16_t *x_filter = av1_get_interp_filter_subpel_kernel(
+ const int16_t *x_filter_ptr = av1_get_interp_filter_subpel_kernel(
filter_params_x, subpel_x_qn & SUBPEL_MASK);
const uint8_t *src_ptr = src - horiz_offset;
- int16_t x_filter_tmp[8];
- int16x8_t filter_x_coef = vld1q_s16(x_filter);
-
- // filter coeffs are even, so downshifting by 1 to reduce intermediate
- // precision requirements.
- filter_x_coef = vshrq_n_s16(filter_x_coef, 1);
- vst1q_s16(&x_filter_tmp[0], filter_x_coef);
+ // Filter values are even, so downshift by 1 to reduce intermediate precision
+ // requirements.
+ const int16x8_t x_filter = vshrq_n_s16(vld1q_s16(x_filter_ptr), 1);
const uint8_t *s;
uint8_t *d_u8;
@@ -980,20 +1235,20 @@
s9 = vget_high_s16(u0);
s10 = vget_high_s16(u1);
- d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter_tmp,
- zero, shift_round_0);
+ d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter, zero,
+ shift_round_0);
d0 = vrshl_s16(d0, horiz_const);
d0 = vadd_s16(d0, round_offset_vec);
- d1 = convolve8_4x4_s16(s1, s2, s3, s4, s5, s6, s7, s8, x_filter_tmp,
- zero, shift_round_0);
+ d1 = convolve8_4x4_s16(s1, s2, s3, s4, s5, s6, s7, s8, x_filter, zero,
+ shift_round_0);
d1 = vrshl_s16(d1, horiz_const);
d1 = vadd_s16(d1, round_offset_vec);
- d2 = convolve8_4x4_s16(s2, s3, s4, s5, s6, s7, s8, s9, x_filter_tmp,
- zero, shift_round_0);
+ d2 = convolve8_4x4_s16(s2, s3, s4, s5, s6, s7, s8, s9, x_filter, zero,
+ shift_round_0);
d2 = vrshl_s16(d2, horiz_const);
d2 = vadd_s16(d2, round_offset_vec);
- d3 = convolve8_4x4_s16(s3, s4, s5, s6, s7, s8, s9, s10, x_filter_tmp,
- zero, shift_round_0);
+ d3 = convolve8_4x4_s16(s3, s4, s5, s6, s7, s8, s9, s10, x_filter, zero,
+ shift_round_0);
d3 = vrshl_s16(d3, horiz_const);
d3 = vadd_s16(d3, round_offset_vec);
@@ -1073,8 +1328,8 @@
s6 = vext_s16(s4, s7, 2); // a6 a7 a8 a9
s7 = vext_s16(s4, s7, 3); // a7 a8 a9 a10
- d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter_tmp,
- zero, shift_round_0);
+ d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter, zero,
+ shift_round_0);
d0 = vrshl_s16(d0, horiz_const);
d0 = vadd_s16(d0, round_offset_vec);
s0 = s4;
@@ -1173,38 +1428,38 @@
s13 = vreinterpretq_s16_u16(vmovl_u8(t6));
s14 = vreinterpretq_s16_u16(vmovl_u8(t7));
- res0 = convolve8_8x8_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter_tmp,
- zero, shift_round_0);
+ res0 = convolve8_8x8_s16(s0, s1, s2, s3, s4, s5, s6, s7, x_filter, zero,
+ shift_round_0);
res0 = vrshlq_s16(res0, horiz_const);
res0 = vaddq_s16(res0, round_offset128);
- res1 = convolve8_8x8_s16(s1, s2, s3, s4, s5, s6, s7, s8, x_filter_tmp,
- zero, shift_round_0);
+ res1 = convolve8_8x8_s16(s1, s2, s3, s4, s5, s6, s7, s8, x_filter, zero,
+ shift_round_0);
res1 = vrshlq_s16(res1, horiz_const);
res1 = vaddq_s16(res1, round_offset128);
- res2 = convolve8_8x8_s16(s2, s3, s4, s5, s6, s7, s8, s9, x_filter_tmp,
- zero, shift_round_0);
+ res2 = convolve8_8x8_s16(s2, s3, s4, s5, s6, s7, s8, s9, x_filter, zero,
+ shift_round_0);
res2 = vrshlq_s16(res2, horiz_const);
res2 = vaddq_s16(res2, round_offset128);
- res3 = convolve8_8x8_s16(s3, s4, s5, s6, s7, s8, s9, s10, x_filter_tmp,
+ res3 = convolve8_8x8_s16(s3, s4, s5, s6, s7, s8, s9, s10, x_filter,
zero, shift_round_0);
res3 = vrshlq_s16(res3, horiz_const);
res3 = vaddq_s16(res3, round_offset128);
- res4 = convolve8_8x8_s16(s4, s5, s6, s7, s8, s9, s10, s11, x_filter_tmp,
+ res4 = convolve8_8x8_s16(s4, s5, s6, s7, s8, s9, s10, s11, x_filter,
zero, shift_round_0);
res4 = vrshlq_s16(res4, horiz_const);
res4 = vaddq_s16(res4, round_offset128);
- res5 = convolve8_8x8_s16(s5, s6, s7, s8, s9, s10, s11, s12,
- x_filter_tmp, zero, shift_round_0);
+ res5 = convolve8_8x8_s16(s5, s6, s7, s8, s9, s10, s11, s12, x_filter,
+ zero, shift_round_0);
res5 = vrshlq_s16(res5, horiz_const);
res5 = vaddq_s16(res5, round_offset128);
- res6 = convolve8_8x8_s16(s6, s7, s8, s9, s10, s11, s12, s13,
- x_filter_tmp, zero, shift_round_0);
+ res6 = convolve8_8x8_s16(s6, s7, s8, s9, s10, s11, s12, s13, x_filter,
+ zero, shift_round_0);
res6 = vrshlq_s16(res6, horiz_const);
res6 = vaddq_s16(res6, round_offset128);
- res7 = convolve8_8x8_s16(s7, s8, s9, s10, s11, s12, s13, s14,
- x_filter_tmp, zero, shift_round_0);
+ res7 = convolve8_8x8_s16(s7, s8, s9, s10, s11, s12, s13, s14, x_filter,
+ zero, shift_round_0);
res7 = vrshlq_s16(res7, horiz_const);
res7 = vaddq_s16(res7, round_offset128);
@@ -1293,8 +1548,8 @@
s6 = vextq_s16(temp_0, s7, 6); // a6 a7 a8 a9 a10 a11 a12 a13
s7 = vextq_s16(temp_0, s7, 7); // a7 a8 a9 a10 a11 a12 a13 a14
- res0 = convolve8_8x8_s16(temp_0, s1, s2, s3, s4, s5, s6, s7,
- x_filter_tmp, zero, shift_round_0);
+ res0 = convolve8_8x8_s16(temp_0, s1, s2, s3, s4, s5, s6, s7, x_filter,
+ zero, shift_round_0);
res0 = vrshlq_s16(res0, horiz_const);
res0 = vaddq_s16(res0, round_offset128);
@@ -1328,6 +1583,8 @@
}
}
+#endif // defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+
void av1_dist_wtd_convolve_y_neon(const uint8_t *src, int src_stride,
uint8_t *dst8, int dst8_stride, int w, int h,
const InterpFilterParams *filter_params_y,
@@ -1352,18 +1609,14 @@
const int shift_value = (conv_params->round_1 - 1 - bits);
// vertical filter
- const int16_t *y_filter = av1_get_interp_filter_subpel_kernel(
+ const int16_t *y_filter_ptr = av1_get_interp_filter_subpel_kernel(
filter_params_y, subpel_y_qn & SUBPEL_MASK);
const uint8_t *src_ptr = src - (vert_offset * src_stride);
- int16_t y_filter_tmp[8];
- int16x8_t filter_y_coef = vld1q_s16(y_filter);
-
- // filter coeffs are even, so downshifting by 1 to reduce intermediate
- // precision requirements.
- filter_y_coef = vshrq_n_s16(filter_y_coef, 1);
- vst1q_s16(&y_filter_tmp[0], filter_y_coef);
+ // Filter values are even, so downshift by 1 to reduce intermediate precision
+ // requirements.
+ const int16x8_t y_filter = vshrq_n_s16(vld1q_s16(y_filter_ptr), 1);
const uint8_t *s;
uint8_t *d_u8;
@@ -1441,17 +1694,17 @@
s9 = vget_low_s16(u1);
s10 = vget_high_s16(u1);
- d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, y_filter_tmp,
- zero, shift_vec);
+ d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, y_filter, zero,
+ shift_vec);
d0 = vadd_s16(d0, round_offset64);
- d1 = convolve8_4x4_s16(s1, s2, s3, s4, s5, s6, s7, s8, y_filter_tmp,
- zero, shift_vec);
+ d1 = convolve8_4x4_s16(s1, s2, s3, s4, s5, s6, s7, s8, y_filter, zero,
+ shift_vec);
d1 = vadd_s16(d1, round_offset64);
- d2 = convolve8_4x4_s16(s2, s3, s4, s5, s6, s7, s8, s9, y_filter_tmp,
- zero, shift_vec);
+ d2 = convolve8_4x4_s16(s2, s3, s4, s5, s6, s7, s8, s9, y_filter, zero,
+ shift_vec);
d2 = vadd_s16(d2, round_offset64);
- d3 = convolve8_4x4_s16(s3, s4, s5, s6, s7, s8, s9, s10, y_filter_tmp,
- zero, shift_vec);
+ d3 = convolve8_4x4_s16(s3, s4, s5, s6, s7, s8, s9, s10, y_filter, zero,
+ shift_vec);
d3 = vadd_s16(d3, round_offset64);
if (conv_params->do_average) {
@@ -1504,8 +1757,8 @@
u0 = vreinterpretq_s16_u16(vmovl_u8(vreinterpret_u8_u32(tu0)));
s7 = vget_low_s16(u0);
- d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, y_filter_tmp,
- zero, shift_vec);
+ d0 = convolve8_4x4_s16(s0, s1, s2, s3, s4, s5, s6, s7, y_filter, zero,
+ shift_vec);
d0 = vadd_s16(d0, round_offset64);
@@ -1602,29 +1855,29 @@
__builtin_prefetch(dst_ptr + 2 * dst_stride);
__builtin_prefetch(dst_ptr + 3 * dst_stride);
- res0 = convolve8_8x8_s16(s0, s1, s2, s3, s4, s5, s6, s7, y_filter_tmp,
- zero, shift_vec);
+ res0 = convolve8_8x8_s16(s0, s1, s2, s3, s4, s5, s6, s7, y_filter, zero,
+ shift_vec);
res0 = vaddq_s16(res0, round_offset128);
- res1 = convolve8_8x8_s16(s1, s2, s3, s4, s5, s6, s7, s8, y_filter_tmp,
- zero, shift_vec);
+ res1 = convolve8_8x8_s16(s1, s2, s3, s4, s5, s6, s7, s8, y_filter, zero,
+ shift_vec);
res1 = vaddq_s16(res1, round_offset128);
- res2 = convolve8_8x8_s16(s2, s3, s4, s5, s6, s7, s8, s9, y_filter_tmp,
- zero, shift_vec);
+ res2 = convolve8_8x8_s16(s2, s3, s4, s5, s6, s7, s8, s9, y_filter, zero,
+ shift_vec);
res2 = vaddq_s16(res2, round_offset128);
- res3 = convolve8_8x8_s16(s3, s4, s5, s6, s7, s8, s9, s10, y_filter_tmp,
+ res3 = convolve8_8x8_s16(s3, s4, s5, s6, s7, s8, s9, s10, y_filter,
zero, shift_vec);
res3 = vaddq_s16(res3, round_offset128);
- res4 = convolve8_8x8_s16(s4, s5, s6, s7, s8, s9, s10, s11, y_filter_tmp,
+ res4 = convolve8_8x8_s16(s4, s5, s6, s7, s8, s9, s10, s11, y_filter,
zero, shift_vec);
res4 = vaddq_s16(res4, round_offset128);
- res5 = convolve8_8x8_s16(s5, s6, s7, s8, s9, s10, s11, s12,
- y_filter_tmp, zero, shift_vec);
+ res5 = convolve8_8x8_s16(s5, s6, s7, s8, s9, s10, s11, s12, y_filter,
+ zero, shift_vec);
res5 = vaddq_s16(res5, round_offset128);
- res6 = convolve8_8x8_s16(s6, s7, s8, s9, s10, s11, s12, s13,
- y_filter_tmp, zero, shift_vec);
+ res6 = convolve8_8x8_s16(s6, s7, s8, s9, s10, s11, s12, s13, y_filter,
+ zero, shift_vec);
res6 = vaddq_s16(res6, round_offset128);
- res7 = convolve8_8x8_s16(s7, s8, s9, s10, s11, s12, s13, s14,
- y_filter_tmp, zero, shift_vec);
+ res7 = convolve8_8x8_s16(s7, s8, s9, s10, s11, s12, s13, s14, y_filter,
+ zero, shift_vec);
res7 = vaddq_s16(res7, round_offset128);
if (conv_params->do_average) {
@@ -1682,8 +1935,8 @@
__builtin_prefetch(dst_ptr);
- res0 = convolve8_8x8_s16(s0, s1, s2, s3, s4, s5, s6, s7, y_filter_tmp,
- zero, shift_vec);
+ res0 = convolve8_8x8_s16(s0, s1, s2, s3, s4, s5, s6, s7, y_filter, zero,
+ shift_vec);
res0 = vaddq_s16(res0, round_offset128);
s0 = s1;
diff --git a/av1/common/av1_rtcd_defs.pl b/av1/common/av1_rtcd_defs.pl
index c9e87e3..333a72d 100644
--- a/av1/common/av1_rtcd_defs.pl
+++ b/av1/common/av1_rtcd_defs.pl
@@ -403,7 +403,7 @@
#
if (aom_config("CONFIG_REALTIME_ONLY") ne "yes") {
add_proto qw/void av1_apply_temporal_filter/, "const struct yv12_buffer_config *ref_frame, const struct macroblockd *mbd, const BLOCK_SIZE block_size, const int mb_row, const int mb_col, const int num_planes, const double *noise_levels, const MV *subblock_mvs, const int *subblock_mses, const int q_factor, const int filter_strength, const uint8_t *pred, uint32_t *accum, uint16_t *count";
- specialize qw/av1_apply_temporal_filter sse2 avx2/;
+ specialize qw/av1_apply_temporal_filter sse2 avx2 neon/;
if (aom_config("CONFIG_AV1_HIGHBITDEPTH") eq "yes") {
add_proto qw/void av1_highbd_apply_temporal_filter/, "const struct yv12_buffer_config *ref_frame, const struct macroblockd *mbd, const BLOCK_SIZE block_size, const int mb_row, const int mb_col, const int num_planes, const double *noise_levels, const MV *subblock_mvs, const int *subblock_mses, const int q_factor, const int filter_strength, const uint8_t *pred, uint32_t *accum, uint16_t *count";
specialize qw/av1_highbd_apply_temporal_filter sse2 avx2/;
@@ -442,7 +442,7 @@
specialize qw/av1_txb_init_levels sse4_1 avx2 neon/;
add_proto qw/uint64_t av1_wedge_sse_from_residuals/, "const int16_t *r1, const int16_t *d, const uint8_t *m, int N";
- specialize qw/av1_wedge_sse_from_residuals sse2 avx2/;
+ specialize qw/av1_wedge_sse_from_residuals sse2 avx2 neon/;
add_proto qw/int8_t av1_wedge_sign_from_residuals/, "const int16_t *ds, const uint8_t *m, int N, int64_t limit";
specialize qw/av1_wedge_sign_from_residuals sse2 avx2/;
add_proto qw/void av1_wedge_compute_delta_squares/, "int16_t *d, const int16_t *a, const int16_t *b, int N";
@@ -450,7 +450,7 @@
# hash
add_proto qw/uint32_t av1_get_crc32c_value/, "void *crc_calculator, uint8_t *p, size_t length";
- specialize qw/av1_get_crc32c_value sse4_2/;
+ specialize qw/av1_get_crc32c_value sse4_2 arm_crc32/;
if (aom_config("CONFIG_REALTIME_ONLY") ne "yes") {
add_proto qw/void av1_compute_stats/, "int wiener_win, const uint8_t *dgd8, const uint8_t *src8, int h_start, int h_end, int v_start, int v_end, int dgd_stride, int src_stride, int64_t *M, int64_t *H, int use_downsampled_wiener_stats";
diff --git a/av1/common/blockd.h b/av1/common/blockd.h
index b2e72d2..6ee02a2 100644
--- a/av1/common/blockd.h
+++ b/av1/common/blockd.h
@@ -188,6 +188,7 @@
typedef struct RD_STATS {
int rate;
+ int zero_rate;
int64_t dist;
// Please be careful of using rdcost, it's not guaranteed to be set all the
// time.
@@ -196,8 +197,7 @@
// rate/dist.
int64_t rdcost;
int64_t sse;
- int skip_txfm; // sse should equal to dist when skip_txfm == 1
- int zero_rate;
+ uint8_t skip_txfm; // sse should equal to dist when skip_txfm == 1
#if CONFIG_RD_DEBUG
int txb_coeff_cost[MAX_MB_PLANE];
#endif // CONFIG_RD_DEBUG
@@ -285,7 +285,7 @@
****************************************************************************/
/**@{*/
/*! \brief Whether to skip transforming and sending. */
- int8_t skip_txfm;
+ uint8_t skip_txfm;
/*! \brief Transform size when fixed size txfm is used (e.g. intra modes). */
TX_SIZE tx_size;
/*! \brief Transform size when recursive txfm tree is on. */
@@ -322,12 +322,11 @@
uint8_t compound_idx : 1;
/*! \brief Whether to use interintra wedge */
uint8_t use_wedge_interintra : 1;
- /*! \brief CDEF strength per BLOCK_64X64 */
- int8_t cdef_strength : 4;
/**@}*/
- /*! \brief Skip CDEF for this superblock */
- uint8_t skip_cdef_curr_sb;
+ /****************************************************************************/
+ /*! \brief CDEF strength per BLOCK_64X64 */
+ int8_t cdef_strength;
#if CONFIG_RD_DEBUG
/*! \brief RD info used for debugging */
@@ -423,12 +422,16 @@
static INLINE int is_global_mv_block(const MB_MODE_INFO *const mbmi,
TransformationType type) {
+ // As global mv is disabled in rt, return from the function before reading
+ // 'mbmi->bsize'. This prevents data race condition in multi-threaded
+ // realtime encoding as mbmi->bsize is updated in the function
+ // direct_partition_merging().
+ if (type <= TRANSLATION) return 0;
const PREDICTION_MODE mode = mbmi->mode;
const BLOCK_SIZE bsize = mbmi->bsize;
const int block_size_allowed =
AOMMIN(block_size_wide[bsize], block_size_high[bsize]) >= 8;
- return (mode == GLOBALMV || mode == GLOBAL_GLOBALMV) && type > TRANSLATION &&
- block_size_allowed;
+ return (mode == GLOBALMV || mode == GLOBAL_GLOBALMV) && block_size_allowed;
}
#if CONFIG_MISMATCH_DEBUG
diff --git a/av1/common/cdef.h b/av1/common/cdef.h
index 5bf40e4..e166f4b 100644
--- a/av1/common/cdef.h
+++ b/av1/common/cdef.h
@@ -89,7 +89,7 @@
* \param[in] xd Pointer to common current coding block structure
* \param[in] cdef_init_fb_row_fn Function Pointer
*
- * \return Nothing is returned. Instead, the filtered frame is output in
+ * \remark Nothing is returned. Instead, the filtered frame is output in
* \c frame.
*/
void av1_cdef_frame(YV12_BUFFER_CONFIG *frame, AV1_COMMON *const cm,
diff --git a/av1/common/convolve.c b/av1/common/convolve.c
index 63dda39..ed77e0d 100644
--- a/av1/common/convolve.c
+++ b/av1/common/convolve.c
@@ -567,28 +567,6 @@
const InterpFilterParams *filter_params_x = interp_filters[0];
const InterpFilterParams *filter_params_y = interp_filters[1];
- // TODO(jingning, yunqing): Add SIMD support to 2-tap filter case.
- // Do we have SIMD support to 4-tap case?
- // 2-tap filter indicates that it is for IntraBC.
- if (filter_params_x->taps == 2 || filter_params_y->taps == 2) {
- assert(filter_params_x->taps == 2 && filter_params_y->taps == 2);
- assert(!scaled);
- if (subpel_x_qn && subpel_y_qn) {
- av1_convolve_2d_sr_c(src, src_stride, dst, dst_stride, w, h,
- filter_params_x, filter_params_y, subpel_x_qn,
- subpel_y_qn, conv_params);
- return;
- } else if (subpel_x_qn) {
- av1_convolve_x_sr_c(src, src_stride, dst, dst_stride, w, h,
- filter_params_x, subpel_x_qn, conv_params);
- return;
- } else if (subpel_y_qn) {
- av1_convolve_y_sr_c(src, src_stride, dst, dst_stride, w, h,
- filter_params_y, subpel_y_qn);
- return;
- }
- }
-
if (scaled) {
convolve_2d_scale_wrapper(src, src_stride, dst, dst_stride, w, h,
filter_params_x, filter_params_y, subpel_x_qn,
diff --git a/av1/common/filter.h b/av1/common/filter.h
index ded5ce5..7511c88 100644
--- a/av1/common/filter.h
+++ b/av1/common/filter.h
@@ -192,14 +192,20 @@
// A special 2-tap bilinear filter for IntraBC chroma. IntraBC uses full pixel
// MV for luma. If sub-sampling exists, chroma may possibly use half-pel MV.
-DECLARE_ALIGNED(256, static const int16_t,
- av1_intrabc_bilinear_filter[2 * SUBPEL_SHIFTS]) = {
- 128, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
- 64, 64, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+DECLARE_ALIGNED(256, static const InterpKernel,
+ av1_intrabc_bilinear_filter[SUBPEL_SHIFTS]) = {
+ { 0, 0, 0, 128, 0, 0, 0, 0 }, { 0, 0, 0, 0, 0, 0, 0, 0 },
+ { 0, 0, 0, 0, 0, 0, 0, 0 }, { 0, 0, 0, 0, 0, 0, 0, 0 },
+ { 0, 0, 0, 0, 0, 0, 0, 0 }, { 0, 0, 0, 0, 0, 0, 0, 0 },
+ { 0, 0, 0, 0, 0, 0, 0, 0 }, { 0, 0, 0, 0, 0, 0, 0, 0 },
+ { 0, 0, 0, 64, 64, 0, 0, 0 }, { 0, 0, 0, 0, 0, 0, 0, 0 },
+ { 0, 0, 0, 0, 0, 0, 0, 0 }, { 0, 0, 0, 0, 0, 0, 0, 0 },
+ { 0, 0, 0, 0, 0, 0, 0, 0 }, { 0, 0, 0, 0, 0, 0, 0, 0 },
+ { 0, 0, 0, 0, 0, 0, 0, 0 }, { 0, 0, 0, 0, 0, 0, 0, 0 },
};
static const InterpFilterParams av1_intrabc_filter_params = {
- av1_intrabc_bilinear_filter, 2, BILINEAR
+ (const int16_t *)av1_intrabc_bilinear_filter, SUBPEL_TAPS, BILINEAR
};
DECLARE_ALIGNED(256, static const InterpKernel,
diff --git a/av1/common/restoration.h b/av1/common/restoration.h
index 65ccd09..a87b9ba 100644
--- a/av1/common/restoration.h
+++ b/av1/common/restoration.h
@@ -397,7 +397,7 @@
* should be at least SGRPROJ_TMPBUF_SIZE big.
* \param[in] optimized_lr Whether to use fast optimized Loop Restoration
*
- * \return Nothing is returned. Instead, the filtered unit is output in
+ * \remark Nothing is returned. Instead, the filtered unit is output in
* \c dst8 at the proper restoration unit offset.
*/
void av1_loop_restoration_filter_unit(
@@ -417,7 +417,7 @@
* \param[in] optimized_lr Whether to use fast optimized Loop Restoration
* \param[in] lr_ctxt Loop restoration context
*
- * \return Nothing is returned. Instead, the filtered frame is output in
+ * \remark Nothing is returned. Instead, the filtered frame is output in
* \c frame.
*/
void av1_loop_restoration_filter_frame(YV12_BUFFER_CONFIG *frame,
diff --git a/av1/common/thread_common.c b/av1/common/thread_common.c
index 54e9443..b951ad3 100644
--- a/av1/common/thread_common.c
+++ b/av1/common/thread_common.c
@@ -845,6 +845,12 @@
copy_funs[plane](lr_ctxt->dst, lr_ctxt->frame, ctxt[plane].tile_rect.left,
ctxt[plane].tile_rect.right, cur_job_info->v_copy_start,
cur_job_info->v_copy_end);
+
+ if (lrworkerdata->do_extend_border) {
+ aom_extend_frame_borders_plane_row(lr_ctxt->frame, plane,
+ cur_job_info->v_copy_start,
+ cur_job_info->v_copy_end);
+ }
} else {
break;
}
@@ -854,7 +860,8 @@
static void foreach_rest_unit_in_planes_mt(AV1LrStruct *lr_ctxt,
AVxWorker *workers, int nworkers,
- AV1LrSync *lr_sync, AV1_COMMON *cm) {
+ AV1LrSync *lr_sync, AV1_COMMON *cm,
+ int do_extend_border) {
FilterFrameCtxt *ctxt = lr_ctxt->ctxt;
const int num_planes = av1_num_planes(cm);
@@ -897,6 +904,7 @@
for (i = num_workers - 1; i >= 0; --i) {
AVxWorker *const worker = &workers[i];
lr_sync->lrworkerdata[i].lr_ctxt = (void *)lr_ctxt;
+ lr_sync->lrworkerdata[i].do_extend_border = do_extend_border;
worker->hook = loop_restoration_row_worker;
worker->data1 = lr_sync;
worker->data2 = &lr_sync->lrworkerdata[i];
@@ -918,7 +926,8 @@
void av1_loop_restoration_filter_frame_mt(YV12_BUFFER_CONFIG *frame,
AV1_COMMON *cm, int optimized_lr,
AVxWorker *workers, int num_workers,
- AV1LrSync *lr_sync, void *lr_ctxt) {
+ AV1LrSync *lr_sync, void *lr_ctxt,
+ int do_extend_border) {
assert(!cm->features.all_lossless);
const int num_planes = av1_num_planes(cm);
@@ -929,7 +938,7 @@
optimized_lr, num_planes);
foreach_rest_unit_in_planes_mt(loop_rest_ctxt, workers, num_workers, lr_sync,
- cm);
+ cm, do_extend_border);
}
// Initializes cdef_sync parameters.
@@ -1002,13 +1011,27 @@
static int cdef_sb_row_worker_hook(void *arg1, void *arg2) {
AV1CdefSync *const cdef_sync = (AV1CdefSync *)arg1;
AV1CdefWorkerData *const cdef_worker = (AV1CdefWorkerData *)arg2;
- const int nvfb =
- (cdef_worker->cm->mi_params.mi_rows + MI_SIZE_64X64 - 1) / MI_SIZE_64X64;
+ AV1_COMMON *cm = cdef_worker->cm;
+ const int nvfb = (cm->mi_params.mi_rows + MI_SIZE_64X64 - 1) / MI_SIZE_64X64;
int cur_fbr;
+ const int num_planes = av1_num_planes(cm);
while (get_cdef_row_next_job(cdef_sync, &cur_fbr, nvfb)) {
- av1_cdef_fb_row(cdef_worker->cm, cdef_worker->xd, cdef_worker->linebuf,
- cdef_worker->colbuf, cdef_worker->srcbuf, cur_fbr,
+ MACROBLOCKD *xd = cdef_worker->xd;
+ av1_cdef_fb_row(cm, xd, cdef_worker->linebuf, cdef_worker->colbuf,
+ cdef_worker->srcbuf, cur_fbr,
cdef_worker->cdef_init_fb_row_fn, cdef_sync);
+ if (cdef_worker->do_extend_border) {
+ for (int plane = 0; plane < num_planes; ++plane) {
+ const YV12_BUFFER_CONFIG *ybf = &cm->cur_frame->buf;
+ const int is_uv = plane > 0;
+ const int mi_high = MI_SIZE_LOG2 - xd->plane[plane].subsampling_y;
+ const int unit_height = MI_SIZE_64X64 << mi_high;
+ const int v_start = cur_fbr * unit_height;
+ const int v_end =
+ AOMMIN(v_start + unit_height, ybf->crop_heights[is_uv]);
+ aom_extend_frame_borders_plane_row(ybf, plane, v_start, v_end);
+ }
+ }
}
return 1;
}
@@ -1017,7 +1040,8 @@
static void prepare_cdef_frame_workers(
AV1_COMMON *const cm, MACROBLOCKD *xd, AV1CdefWorkerData *const cdef_worker,
AVxWorkerHook hook, AVxWorker *const workers, AV1CdefSync *const cdef_sync,
- int num_workers, cdef_init_fb_row_t cdef_init_fb_row_fn) {
+ int num_workers, cdef_init_fb_row_t cdef_init_fb_row_fn,
+ int do_extend_border) {
const int num_planes = av1_num_planes(cm);
cdef_worker[0].srcbuf = cm->cdef_info.srcbuf;
@@ -1028,6 +1052,7 @@
cdef_worker[i].cm = cm;
cdef_worker[i].xd = xd;
cdef_worker[i].cdef_init_fb_row_fn = cdef_init_fb_row_fn;
+ cdef_worker[i].do_extend_border = do_extend_border;
for (int plane = 0; plane < num_planes; plane++)
cdef_worker[i].linebuf[plane] = cm->cdef_info.linebuf[plane];
@@ -1111,8 +1136,8 @@
void av1_cdef_frame_mt(AV1_COMMON *const cm, MACROBLOCKD *const xd,
AV1CdefWorkerData *const cdef_worker,
AVxWorker *const workers, AV1CdefSync *const cdef_sync,
- int num_workers,
- cdef_init_fb_row_t cdef_init_fb_row_fn) {
+ int num_workers, cdef_init_fb_row_t cdef_init_fb_row_fn,
+ int do_extend_border) {
YV12_BUFFER_CONFIG *frame = &cm->cur_frame->buf;
const int num_planes = av1_num_planes(cm);
@@ -1122,7 +1147,7 @@
reset_cdef_job_info(cdef_sync);
prepare_cdef_frame_workers(cm, xd, cdef_worker, cdef_sb_row_worker_hook,
workers, cdef_sync, num_workers,
- cdef_init_fb_row_fn);
+ cdef_init_fb_row_fn, do_extend_border);
launch_cdef_workers(workers, num_workers);
sync_cdef_workers(workers, cm, num_workers);
}
diff --git a/av1/common/thread_common.h b/av1/common/thread_common.h
index 7c284fa..b1e622f 100644
--- a/av1/common/thread_common.h
+++ b/av1/common/thread_common.h
@@ -70,6 +70,7 @@
int32_t *rst_tmpbuf;
void *rlbs;
void *lr_ctxt;
+ int do_extend_border;
} LRWorkerData;
// Looprestoration row synchronization
@@ -106,6 +107,7 @@
uint16_t *srcbuf;
uint16_t *linebuf[MAX_MB_PLANE];
cdef_init_fb_row_t cdef_init_fb_row_fn;
+ int do_extend_border;
} AV1CdefWorkerData;
typedef struct AV1CdefRowSync {
@@ -135,7 +137,8 @@
void av1_cdef_frame_mt(AV1_COMMON *const cm, MACROBLOCKD *const xd,
AV1CdefWorkerData *const cdef_worker,
AVxWorker *const workers, AV1CdefSync *const cdef_sync,
- int num_workers, cdef_init_fb_row_t cdef_init_fb_row_fn);
+ int num_workers, cdef_init_fb_row_t cdef_init_fb_row_fn,
+ int do_extend_border);
void av1_cdef_init_fb_row_mt(const AV1_COMMON *const cm,
const MACROBLOCKD *const xd,
CdefBlockInfo *const fb_info,
@@ -163,7 +166,7 @@
struct AV1Common *cm,
int optimized_lr, AVxWorker *workers,
int num_workers, AV1LrSync *lr_sync,
- void *lr_ctxt);
+ void *lr_ctxt, int do_extend_border);
void av1_loop_restoration_dealloc(AV1LrSync *lr_sync, int num_workers);
void av1_loop_restoration_alloc(AV1LrSync *lr_sync, AV1_COMMON *cm,
int num_workers, int num_rows_lr,
diff --git a/av1/common/txb_common.h b/av1/common/txb_common.h
index 5ba3951..40fcffc 100644
--- a/av1/common/txb_common.h
+++ b/av1/common/txb_common.h
@@ -78,71 +78,6 @@
return idx + ((idx >> bwl) << TX_PAD_HOR_LOG2);
}
-static INLINE int get_base_ctx_from_count_mag(int row, int col, int count,
- int sig_mag) {
- const int ctx = base_level_count_to_index[count];
- int ctx_idx = -1;
-
- if (row == 0 && col == 0) {
- if (sig_mag >= 2) return 0;
-
- if (sig_mag == 1) {
- if (count >= 2)
- ctx_idx = 1;
- else
- ctx_idx = 2;
-
- return ctx_idx;
- }
-
- ctx_idx = 3 + ctx;
- assert(ctx_idx <= 6);
- return ctx_idx;
- } else if (row == 0) {
- if (sig_mag >= 2) return 6;
- if (sig_mag == 1) {
- if (count >= 2)
- ctx_idx = 7;
- else
- ctx_idx = 8;
- return ctx_idx;
- }
-
- ctx_idx = 9 + ctx;
- assert(ctx_idx <= 11);
- return ctx_idx;
- } else if (col == 0) {
- if (sig_mag >= 2) return 12;
- if (sig_mag == 1) {
- if (count >= 2)
- ctx_idx = 13;
- else
- ctx_idx = 14;
-
- return ctx_idx;
- }
-
- ctx_idx = 15 + ctx;
- assert(ctx_idx <= 17);
- // TODO(angiebird): turn this on once the optimization is finalized
- // assert(ctx_idx < 28);
- } else {
- if (sig_mag >= 2) return 18;
- if (sig_mag == 1) {
- if (count >= 2)
- ctx_idx = 19;
- else
- ctx_idx = 20;
- return ctx_idx;
- }
-
- ctx_idx = 21 + ctx;
-
- assert(ctx_idx <= 24);
- }
- return ctx_idx;
-}
-
static INLINE int get_br_ctx_2d(const uint8_t *const levels,
const int c, // raster order
const int bwl) {
@@ -351,11 +286,11 @@
*cul_level += 2 << COEFF_CONTEXT_BITS;
}
-static INLINE void get_txb_ctx(const BLOCK_SIZE plane_bsize,
- const TX_SIZE tx_size, const int plane,
- const ENTROPY_CONTEXT *const a,
- const ENTROPY_CONTEXT *const l,
- TXB_CTX *const txb_ctx) {
+static void get_txb_ctx_general(const BLOCK_SIZE plane_bsize,
+ const TX_SIZE tx_size, const int plane,
+ const ENTROPY_CONTEXT *const a,
+ const ENTROPY_CONTEXT *const l,
+ TXB_CTX *const txb_ctx) {
#define MAX_TX_SIZE_UNIT 16
static const int8_t signs[3] = { 0, -1, 1 };
static const int8_t dc_sign_contexts[4 * MAX_TX_SIZE_UNIT + 1] = {
@@ -437,7 +372,100 @@
: 7;
txb_ctx->txb_skip_ctx = ctx_base + ctx_offset;
}
-#undef MAX_TX_SIZE_UNIT
}
+#define SPECIALIZE_GET_TXB_CTX(w, h) \
+ static void get_txb_ctx_##w##x##h( \
+ const BLOCK_SIZE plane_bsize, const int plane, \
+ const ENTROPY_CONTEXT *const a, const ENTROPY_CONTEXT *const l, \
+ TXB_CTX *const txb_ctx) { \
+ static const int8_t signs[3] = { 0, -1, 1 }; \
+ static const int8_t dc_sign_contexts[4 * MAX_TX_SIZE_UNIT + 1] = { \
+ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \
+ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, \
+ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2 \
+ }; \
+ const TX_SIZE tx_size = TX_##w##X##h; \
+ const int txb_w_unit = tx_size_wide_unit[tx_size]; \
+ const int txb_h_unit = tx_size_high_unit[tx_size]; \
+ int dc_sign = 0; \
+ int k = 0; \
+ \
+ do { \
+ const unsigned int sign = ((uint8_t)a[k]) >> COEFF_CONTEXT_BITS; \
+ assert(sign <= 2); \
+ dc_sign += signs[sign]; \
+ } while (++k < txb_w_unit); \
+ \
+ k = 0; \
+ do { \
+ const unsigned int sign = ((uint8_t)l[k]) >> COEFF_CONTEXT_BITS; \
+ assert(sign <= 2); \
+ dc_sign += signs[sign]; \
+ } while (++k < txb_h_unit); \
+ \
+ txb_ctx->dc_sign_ctx = dc_sign_contexts[dc_sign + 2 * MAX_TX_SIZE_UNIT]; \
+ \
+ if (plane == 0) { \
+ if (plane_bsize == txsize_to_bsize[tx_size]) { \
+ txb_ctx->txb_skip_ctx = 0; \
+ } else { \
+ static const uint8_t skip_contexts[5][5] = { { 1, 2, 2, 2, 3 }, \
+ { 2, 4, 4, 4, 5 }, \
+ { 2, 4, 4, 4, 5 }, \
+ { 2, 4, 4, 4, 5 }, \
+ { 3, 5, 5, 5, 6 } }; \
+ int top = 0; \
+ int left = 0; \
+ \
+ k = 0; \
+ do { \
+ top |= a[k]; \
+ } while (++k < txb_w_unit); \
+ top &= COEFF_CONTEXT_MASK; \
+ top = AOMMIN(top, 4); \
+ \
+ k = 0; \
+ do { \
+ left |= l[k]; \
+ } while (++k < txb_h_unit); \
+ left &= COEFF_CONTEXT_MASK; \
+ left = AOMMIN(left, 4); \
+ \
+ txb_ctx->txb_skip_ctx = skip_contexts[top][left]; \
+ } \
+ } else { \
+ const int ctx_base = get_entropy_context(tx_size, a, l); \
+ const int ctx_offset = (num_pels_log2_lookup[plane_bsize] > \
+ num_pels_log2_lookup[txsize_to_bsize[tx_size]]) \
+ ? 10 \
+ : 7; \
+ txb_ctx->txb_skip_ctx = ctx_base + ctx_offset; \
+ } \
+ }
+
+SPECIALIZE_GET_TXB_CTX(4, 4)
+SPECIALIZE_GET_TXB_CTX(8, 8)
+SPECIALIZE_GET_TXB_CTX(16, 16)
+SPECIALIZE_GET_TXB_CTX(32, 32)
+
+// Wrapper for get_txb_ctx that calls the specialized version of get_txb_ctc_*
+// so that the compiler can compile away the while loops.
+static INLINE void get_txb_ctx(const BLOCK_SIZE plane_bsize,
+ const TX_SIZE tx_size, const int plane,
+ const ENTROPY_CONTEXT *const a,
+ const ENTROPY_CONTEXT *const l,
+ TXB_CTX *const txb_ctx) {
+ switch (tx_size) {
+ case TX_4X4: get_txb_ctx_4x4(plane_bsize, plane, a, l, txb_ctx); break;
+ case TX_8X8: get_txb_ctx_8x8(plane_bsize, plane, a, l, txb_ctx); break;
+ case TX_16X16: get_txb_ctx_16x16(plane_bsize, plane, a, l, txb_ctx); break;
+ case TX_32X32: get_txb_ctx_32x32(plane_bsize, plane, a, l, txb_ctx); break;
+ default:
+ get_txb_ctx_general(plane_bsize, tx_size, plane, a, l, txb_ctx);
+ break;
+ }
+}
+#undef MAX_TX_SIZE_UNIT
+
#endif // AOM_AV1_COMMON_TXB_COMMON_H_
diff --git a/av1/common/x86/av1_convolve_scale_sse4.c b/av1/common/x86/av1_convolve_scale_sse4.c
index 0bdf49f..67b28bc 100644
--- a/av1/common/x86/av1_convolve_scale_sse4.c
+++ b/av1/common/x86/av1_convolve_scale_sse4.c
@@ -22,7 +22,7 @@
// av1_convolve_2d_scale_sse4_1. This version only supports 8 tap filters.
static void hfilter8(const uint8_t *src, int src_stride, int16_t *dst, int w,
int h, int subpel_x_qn, int x_step_qn,
- const InterpFilterParams *filter_params, unsigned round) {
+ const InterpFilterParams *filter_params, int round) {
const int bd = 8;
const int ntaps = 8;
@@ -168,11 +168,11 @@
_mm_sra_epi32(_mm_add_epi32(conv, round_shift_add), round_shift);
uint8_t *dst_x = dst + y * dst_stride + x;
- CONV_BUF_TYPE *dst_16_x = dst16 + y * dst16_stride + x;
__m128i result;
__m128i shifted_16 = _mm_packus_epi32(shifted, shifted);
if (conv_params->is_compound) {
+ CONV_BUF_TYPE *dst_16_x = dst16 + y * dst16_stride + x;
if (conv_params->do_average) {
const __m128i p_16 = _mm_loadl_epi64((__m128i *)dst_16_x);
if (conv_params->use_dist_wtd_comp_avg) {
@@ -260,8 +260,8 @@
// filters.
static void highbd_hfilter8(const uint16_t *src, int src_stride, int16_t *dst,
int w, int h, int subpel_x_qn, int x_step_qn,
- const InterpFilterParams *filter_params,
- unsigned round, int bd) {
+ const InterpFilterParams *filter_params, int round,
+ int bd) {
const int ntaps = 8;
src -= ntaps / 2 - 1;
@@ -399,10 +399,10 @@
_mm_sra_epi32(_mm_add_epi32(conv, round_shift_add), round_shift);
uint16_t *dst_x = dst + y * dst_stride + x;
- CONV_BUF_TYPE *dst_16_x = dst16 + y * dst16_stride + x;
__m128i result;
if (conv_params->is_compound) {
+ CONV_BUF_TYPE *dst_16_x = dst16 + y * dst16_stride + x;
if (conv_params->do_average) {
__m128i p_32 =
_mm_cvtepu16_epi32(_mm_loadl_epi64((__m128i *)dst_16_x));
@@ -414,20 +414,20 @@
} else {
shifted = _mm_srai_epi32(_mm_add_epi32(p_32, shifted), 1);
}
- __m128i res32 = _mm_sub_epi32(shifted, sub);
- res32 = _mm_sra_epi32(_mm_add_epi32(res32, round_bits_const),
- round_bits_shift);
+ result = _mm_sub_epi32(shifted, sub);
+ result = _mm_sra_epi32(_mm_add_epi32(result, round_bits_const),
+ round_bits_shift);
- __m128i res16 = _mm_packus_epi32(res32, res32);
- res16 = _mm_min_epi16(res16, clip_pixel_);
- _mm_storel_epi64((__m128i *)dst_x, res16);
+ result = _mm_packus_epi32(result, result);
+ result = _mm_min_epi16(result, clip_pixel_);
+ _mm_storel_epi64((__m128i *)dst_x, result);
} else {
__m128i shifted_16 = _mm_packus_epi32(shifted, shifted);
_mm_storel_epi64((__m128i *)dst_16_x, shifted_16);
}
} else {
- const __m128i subbed = _mm_sub_epi32(shifted, sub);
- result = _mm_sra_epi16(_mm_add_epi32(subbed, bits_const), bits_shift);
+ result = _mm_sub_epi32(shifted, sub);
+ result = _mm_sra_epi16(_mm_add_epi32(result, bits_const), bits_shift);
result = _mm_packus_epi32(result, result);
result = _mm_min_epi16(result, clip_pixel_);
_mm_storel_epi64((__m128i *)dst_x, result);
diff --git a/av1/common/x86/av1_inv_txfm_ssse3.c b/av1/common/x86/av1_inv_txfm_ssse3.c
index f9bfb37..738cc98 100644
--- a/av1/common/x86/av1_inv_txfm_ssse3.c
+++ b/av1/common/x86/av1_inv_txfm_ssse3.c
@@ -2246,7 +2246,7 @@
const int step = flipud ? -1 : 1;
const __m128i zero = _mm_setzero_si128();
for (int i = 0; i < height; ++i, j += step) {
- const __m128i v = _mm_cvtsi32_si128(*((uint32_t *)(output + i * stride)));
+ const __m128i v = _mm_cvtsi32_si128(*((int *)(output + i * stride)));
__m128i u = _mm_adds_epi16(in[j], _mm_unpacklo_epi8(v, zero));
u = _mm_packus_epi16(u, zero);
*((int *)(output + i * stride)) = _mm_cvtsi128_si32(u);
diff --git a/av1/common/x86/convolve_sse2.c b/av1/common/x86/convolve_sse2.c
index 0e77822..012e75c 100644
--- a/av1/common/x86/convolve_sse2.c
+++ b/av1/common/x86/convolve_sse2.c
@@ -200,31 +200,31 @@
if (w <= 4) {
__m128i s[8], src6, res, res_round, res16;
int res_int;
- src6 = _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 6 * src_stride));
+ src6 = _mm_cvtsi32_si128(*(int *)(src_ptr + 6 * src_stride));
s[0] = _mm_unpacklo_epi8(
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 0 * src_stride)),
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 1 * src_stride)));
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 0 * src_stride)),
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 1 * src_stride)));
s[1] = _mm_unpacklo_epi8(
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 1 * src_stride)),
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 2 * src_stride)));
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 1 * src_stride)),
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 2 * src_stride)));
s[2] = _mm_unpacklo_epi8(
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 2 * src_stride)),
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 3 * src_stride)));
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 2 * src_stride)),
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 3 * src_stride)));
s[3] = _mm_unpacklo_epi8(
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 3 * src_stride)),
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 4 * src_stride)));
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 3 * src_stride)),
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 4 * src_stride)));
s[4] = _mm_unpacklo_epi8(
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 4 * src_stride)),
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 5 * src_stride)));
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 4 * src_stride)),
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 5 * src_stride)));
s[5] = _mm_unpacklo_epi8(
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 5 * src_stride)), src6);
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 5 * src_stride)), src6);
do {
s[6] = _mm_unpacklo_epi8(
- src6, _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 7 * src_stride)));
- src6 = _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 8 * src_stride));
+ src6, _mm_cvtsi32_si128(*(int *)(src_ptr + 7 * src_stride)));
+ src6 = _mm_cvtsi32_si128(*(int *)(src_ptr + 8 * src_stride));
s[7] = _mm_unpacklo_epi8(
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 7 * src_stride)), src6);
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 7 * src_stride)), src6);
res = convolve_lo_y(s + 0, coeffs);
res_round = _mm_sra_epi32(_mm_add_epi32(res, round_const), round_shift);
diff --git a/av1/common/x86/highbd_convolve_2d_avx2.c b/av1/common/x86/highbd_convolve_2d_avx2.c
index 12046e4..429f8f1 100644
--- a/av1/common/x86/highbd_convolve_2d_avx2.c
+++ b/av1/common/x86/highbd_convolve_2d_avx2.c
@@ -80,7 +80,7 @@
for (i = 0; i < im_h; i += 2) {
const __m256i row0 =
_mm256_loadu_si256((__m256i *)&src_ptr[i * src_stride + j]);
- __m256i row1 = _mm256_set1_epi16(0);
+ __m256i row1 = _mm256_setzero_si256();
if (i + 1 < im_h)
row1 =
_mm256_loadu_si256((__m256i *)&src_ptr[(i + 1) * src_stride + j]);
diff --git a/av1/common/x86/highbd_inv_txfm_sse4.c b/av1/common/x86/highbd_inv_txfm_sse4.c
index 37f8f42..de3af3a 100644
--- a/av1/common/x86/highbd_inv_txfm_sse4.c
+++ b/av1/common/x86/highbd_inv_txfm_sse4.c
@@ -517,7 +517,7 @@
static void iadst4x4_sse4_1(__m128i *in, __m128i *out, int bit, int do_cols,
int bd, int out_shift) {
const int32_t *sinpi = sinpi_arr(bit);
- const __m128i zero = _mm_set1_epi32(0);
+ const __m128i zero = _mm_setzero_si128();
__m128i rnding = _mm_set1_epi32(1 << (bit + 4 - 1));
rnding = _mm_unpacklo_epi32(rnding, zero);
const __m128i mul = _mm_set1_epi32(1 << 4);
@@ -698,7 +698,7 @@
int bd, int out_shift) {
(void)bit;
__m128i v[4];
- __m128i zero = _mm_set1_epi32(0);
+ __m128i zero = _mm_setzero_si128();
__m128i fact = _mm_set1_epi32(NewSqrt2);
__m128i offset = _mm_set1_epi32(1 << (NewSqrt2Bits - 1));
__m128i a0_low, a1_low;
@@ -3142,7 +3142,7 @@
__m128i fact = _mm_set1_epi32(2 * NewSqrt2);
__m128i offset = _mm_set1_epi32(1 << (NewSqrt2Bits - 1));
__m128i a0_low, a0_high, a1_low, a1_high;
- __m128i zero = _mm_set1_epi32(0);
+ __m128i zero = _mm_setzero_si128();
offset = _mm_unpacklo_epi32(offset, zero);
for (int i = 0; i < 16; i++) {
diff --git a/av1/common/x86/highbd_jnt_convolve_avx2.c b/av1/common/x86/highbd_jnt_convolve_avx2.c
index 9cedd44..da52ecd 100644
--- a/av1/common/x86/highbd_jnt_convolve_avx2.c
+++ b/av1/common/x86/highbd_jnt_convolve_avx2.c
@@ -282,7 +282,7 @@
for (i = 0; i < im_h; i += 2) {
const __m256i row0 =
_mm256_loadu_si256((__m256i *)&src_ptr[i * src_stride + j]);
- __m256i row1 = _mm256_set1_epi16(0);
+ __m256i row1 = _mm256_setzero_si256();
if (i + 1 < im_h)
row1 =
_mm256_loadu_si256((__m256i *)&src_ptr[(i + 1) * src_stride + j]);
diff --git a/av1/common/x86/highbd_warp_affine_avx2.c b/av1/common/x86/highbd_warp_affine_avx2.c
index 87b1a66..7f6aceb 100644
--- a/av1/common/x86/highbd_warp_affine_avx2.c
+++ b/av1/common/x86/highbd_warp_affine_avx2.c
@@ -158,7 +158,7 @@
iy = iy * stride;
__m256i v_refl = _mm256_inserti128_si256(
- _mm256_set1_epi16(0),
+ _mm256_setzero_si256(),
_mm_loadu_si128((__m128i *)&ref[iy + ix4 - 7]), 0);
v_refl = _mm256_inserti128_si256(
v_refl, _mm_loadu_si128((__m128i *)&ref[iy + ix4 + 1]),
@@ -218,7 +218,7 @@
_mm_shuffle_epi32(v_01, 3)); // A7A6A7A6A7A6A7A6
__m256i v_refl = _mm256_inserti128_si256(
- _mm256_set1_epi16(0),
+ _mm256_setzero_si256(),
_mm_loadu_si128((__m128i *)&ref[iy + ix4 - 7]), 0);
v_refl = _mm256_inserti128_si256(
v_refl, _mm_loadu_si128((__m128i *)&ref[iy + ix4 + 1]),
@@ -333,7 +333,7 @@
iy = iy * stride;
__m256i v_refl = _mm256_inserti128_si256(
- _mm256_set1_epi16(0),
+ _mm256_setzero_si256(),
_mm_loadu_si128((__m128i *)&ref[iy + ix4 - 7]), 0);
v_refl = _mm256_inserti128_si256(
v_refl, _mm_loadu_si128((__m128i *)&ref[iy + ix4 + 1]),
@@ -454,7 +454,7 @@
_mm256_unpackhi_epi64(v_c0123u, v_c4567u); // H7H6 ... A7A6
__m256i v_refl = _mm256_inserti128_si256(
- _mm256_set1_epi16(0),
+ _mm256_setzero_si256(),
_mm_loadu_si128((__m128i *)&ref[iy + ix4 - 7]), 0);
v_refl = _mm256_inserti128_si256(
v_refl, _mm_loadu_si128((__m128i *)&ref[iy + ix4 + 1]),
diff --git a/av1/common/x86/intra_edge_sse4.c b/av1/common/x86/intra_edge_sse4.c
index fc69f41..f025f79 100644
--- a/av1/common/x86/intra_edge_sse4.c
+++ b/av1/common/x86/intra_edge_sse4.c
@@ -33,7 +33,7 @@
// Extend the first and last samples to simplify the loop for the 5-tap case
p[-1] = p[0];
- __m128i last = _mm_set1_epi8(p[sz - 1]);
+ __m128i last = _mm_set1_epi8((char)p[sz - 1]);
_mm_storeu_si128((__m128i *)&p[sz], last);
// Adjust input pointer for filter support area
diff --git a/av1/common/x86/jnt_convolve_sse2.c b/av1/common/x86/jnt_convolve_sse2.c
index 581d150..ab937f9 100644
--- a/av1/common/x86/jnt_convolve_sse2.c
+++ b/av1/common/x86/jnt_convolve_sse2.c
@@ -178,31 +178,31 @@
if (w == 4) {
__m128i s[8], src6, res, res_shift;
- src6 = _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 6 * src_stride));
+ src6 = _mm_cvtsi32_si128(*(int *)(src_ptr + 6 * src_stride));
s[0] = _mm_unpacklo_epi8(
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 0 * src_stride)),
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 1 * src_stride)));
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 0 * src_stride)),
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 1 * src_stride)));
s[1] = _mm_unpacklo_epi8(
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 1 * src_stride)),
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 2 * src_stride)));
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 1 * src_stride)),
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 2 * src_stride)));
s[2] = _mm_unpacklo_epi8(
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 2 * src_stride)),
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 3 * src_stride)));
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 2 * src_stride)),
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 3 * src_stride)));
s[3] = _mm_unpacklo_epi8(
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 3 * src_stride)),
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 4 * src_stride)));
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 3 * src_stride)),
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 4 * src_stride)));
s[4] = _mm_unpacklo_epi8(
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 4 * src_stride)),
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 5 * src_stride)));
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 4 * src_stride)),
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 5 * src_stride)));
s[5] = _mm_unpacklo_epi8(
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 5 * src_stride)), src6);
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 5 * src_stride)), src6);
do {
s[6] = _mm_unpacklo_epi8(
- src6, _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 7 * src_stride)));
- src6 = _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 8 * src_stride));
+ src6, _mm_cvtsi32_si128(*(int *)(src_ptr + 7 * src_stride)));
+ src6 = _mm_cvtsi32_si128(*(int *)(src_ptr + 8 * src_stride));
s[7] = _mm_unpacklo_epi8(
- _mm_cvtsi32_si128(*(uint32_t *)(src_ptr + 7 * src_stride)), src6);
+ _mm_cvtsi32_si128(*(int *)(src_ptr + 7 * src_stride)), src6);
res = convolve_lo_y(s + 0, coeffs);
res_shift = _mm_sll_epi32(res, left_shift);
diff --git a/av1/common/x86/reconinter_sse4.c b/av1/common/x86/reconinter_sse4.c
index a503532..95814b4 100644
--- a/av1/common/x86/reconinter_sse4.c
+++ b/av1/common/x86/reconinter_sse4.c
@@ -33,13 +33,13 @@
int i = 0;
if (4 == w) {
do {
- const __m128i s0A = _mm_cvtsi32_si128(*(uint32_t *)src0);
- const __m128i s0B = _mm_cvtsi32_si128(*(uint32_t *)(src0 + stride0));
+ const __m128i s0A = _mm_cvtsi32_si128(*(int *)src0);
+ const __m128i s0B = _mm_cvtsi32_si128(*(int *)(src0 + stride0));
const __m128i s0AB = _mm_unpacklo_epi32(s0A, s0B);
const __m128i s0 = _mm_cvtepu8_epi16(s0AB);
- const __m128i s1A = _mm_cvtsi32_si128(*(uint32_t *)src1);
- const __m128i s1B = _mm_cvtsi32_si128(*(uint32_t *)(src1 + stride1));
+ const __m128i s1A = _mm_cvtsi32_si128(*(int *)src1);
+ const __m128i s1B = _mm_cvtsi32_si128(*(int *)(src1 + stride1));
const __m128i s1AB = _mm_unpacklo_epi32(s1A, s1B);
const __m128i s1 = _mm_cvtepu8_epi16(s1AB);
diff --git a/av1/common/x86/selfguided_avx2.c b/av1/common/x86/selfguided_avx2.c
index 3c5558d..4ab35e8 100644
--- a/av1/common/x86/selfguided_avx2.c
+++ b/av1/common/x86/selfguided_avx2.c
@@ -230,7 +230,7 @@
const __m256i rnd_res = round_for_shift(SGRPROJ_RECIP_BITS);
// Set up masks
- const __m128i ones32 = _mm_set_epi32(0, 0, 0xffffffff, 0xffffffff);
+ const __m128i ones32 = _mm_set_epi32(0, 0, ~0, ~0);
__m256i mask[8];
for (int idx = 0; idx < 8; idx++) {
const __m128i shift = _mm_cvtsi32_si128(8 * (8 - idx));
@@ -367,7 +367,7 @@
const __m256i rnd_res = round_for_shift(SGRPROJ_RECIP_BITS);
// Set up masks
- const __m128i ones32 = _mm_set_epi32(0, 0, 0xffffffff, 0xffffffff);
+ const __m128i ones32 = _mm_set_epi32(0, 0, ~0, ~0);
__m256i mask[8];
for (int idx = 0; idx < 8; idx++) {
const __m128i shift = _mm_cvtsi32_si128(8 * (8 - idx));
diff --git a/av1/common/x86/selfguided_sse4.c b/av1/common/x86/selfguided_sse4.c
index 72c7708..948bbfb 100644
--- a/av1/common/x86/selfguided_sse4.c
+++ b/av1/common/x86/selfguided_sse4.c
@@ -181,7 +181,7 @@
const __m128i rnd_res = round_for_shift(SGRPROJ_RECIP_BITS);
// Set up masks
- const __m128i ones32 = _mm_set_epi32(0, 0, 0xffffffff, 0xffffffff);
+ const __m128i ones32 = _mm_set_epi32(0, 0, ~0, ~0);
__m128i mask[4];
for (int idx = 0; idx < 4; idx++) {
const __m128i shift = _mm_cvtsi32_si128(8 * (4 - idx));
@@ -322,7 +322,7 @@
const __m128i rnd_res = round_for_shift(SGRPROJ_RECIP_BITS);
// Set up masks
- const __m128i ones32 = _mm_set_epi32(0, 0, 0xffffffff, 0xffffffff);
+ const __m128i ones32 = _mm_set_epi32(0, 0, ~0, ~0);
__m128i mask[4];
for (int idx = 0; idx < 4; idx++) {
const __m128i shift = _mm_cvtsi32_si128(8 * (4 - idx));
diff --git a/av1/common/x86/warp_plane_avx2.c b/av1/common/x86/warp_plane_avx2.c
index b0c9a93..ceb836e 100644
--- a/av1/common/x86/warp_plane_avx2.c
+++ b/av1/common/x86/warp_plane_avx2.c
@@ -1028,12 +1028,12 @@
int64_t sum_error = 0;
int i, j;
__m256i row_error, col_error;
- __m256i zero = _mm256_set1_epi16(0);
+ __m256i zero = _mm256_setzero_si256();
__m256i dup_255 = _mm256_set1_epi16(255);
col_error = zero;
for (i = 0; i < (p_height / 4); i++) {
- row_error = _mm256_set1_epi16(0);
+ row_error = _mm256_setzero_si256();
for (j = 0; j < (p_width / 16); j++) {
__m256i ref_1_16 = _mm256_cvtepu8_epi16(_mm_load_si128(
(__m128i *)(ref + (j * 16) + (((i * 4) + 0) * ref_stride))));
diff --git a/av1/common/x86/warp_plane_sse2.c b/av1/common/x86/warp_plane_sse2.c
index 6ff6665..f8fe578 100644
--- a/av1/common/x86/warp_plane_sse2.c
+++ b/av1/common/x86/warp_plane_sse2.c
@@ -21,7 +21,7 @@
int64_t sum_error = 0;
int i, j;
__m128i row_error, col_error;
- __m128i zero = _mm_set1_epi16(0);
+ __m128i zero = _mm_setzero_si128();
__m128i dup_255 = _mm_set1_epi16(255);
col_error = zero;
for (i = 0; i < (p_height); i++) {
diff --git a/av1/decoder/decodeframe.c b/av1/decoder/decodeframe.c
index 8e7ffce..34dd438 100644
--- a/av1/decoder/decodeframe.c
+++ b/av1/decoder/decodeframe.c
@@ -5280,6 +5280,9 @@
cm->rst_info[0].frame_restoration_type != RESTORE_NONE ||
cm->rst_info[1].frame_restoration_type != RESTORE_NONE ||
cm->rst_info[2].frame_restoration_type != RESTORE_NONE;
+ // Frame border extension is not required in the decoder
+ // as it happens in extend_mc_border().
+ int do_extend_border_mt = 0;
if (!optimized_loop_restoration) {
if (do_loop_restoration)
av1_loop_restoration_save_boundary_lines(&pbi->common.cur_frame->buf,
@@ -5289,7 +5292,8 @@
if (pbi->num_workers > 1) {
av1_cdef_frame_mt(cm, &pbi->dcb.xd, pbi->cdef_worker,
pbi->tile_workers, &pbi->cdef_sync,
- pbi->num_workers, av1_cdef_init_fb_row_mt);
+ pbi->num_workers, av1_cdef_init_fb_row_mt,
+ do_extend_border_mt);
} else {
av1_cdef_frame(&pbi->common.cur_frame->buf, cm, &pbi->dcb.xd,
av1_cdef_init_fb_row);
@@ -5305,7 +5309,7 @@
av1_loop_restoration_filter_frame_mt(
(YV12_BUFFER_CONFIG *)xd->cur_buf, cm, optimized_loop_restoration,
pbi->tile_workers, pbi->num_workers, &pbi->lr_row_sync,
- &pbi->lr_ctxt);
+ &pbi->lr_ctxt, do_extend_border_mt);
} else {
av1_loop_restoration_filter_frame((YV12_BUFFER_CONFIG *)xd->cur_buf,
cm, optimized_loop_restoration,
@@ -5320,7 +5324,7 @@
av1_loop_restoration_filter_frame_mt(
(YV12_BUFFER_CONFIG *)xd->cur_buf, cm, optimized_loop_restoration,
pbi->tile_workers, pbi->num_workers, &pbi->lr_row_sync,
- &pbi->lr_ctxt);
+ &pbi->lr_ctxt, do_extend_border_mt);
} else {
av1_loop_restoration_filter_frame((YV12_BUFFER_CONFIG *)xd->cur_buf,
cm, optimized_loop_restoration,
diff --git a/av1/decoder/decodemv.c b/av1/decoder/decodemv.c
index 6e7291d..1988eb2 100644
--- a/av1/decoder/decodemv.c
+++ b/av1/decoder/decodemv.c
@@ -1204,7 +1204,9 @@
.as_int;
break;
}
- default: { return 0; }
+ default: {
+ return 0;
+ }
}
int ret = is_mv_valid(&mv[0].as_mv);
diff --git a/av1/decoder/decoder.c b/av1/decoder/decoder.c
index 2553ffb..f1ffaa4 100644
--- a/av1/decoder/decoder.c
+++ b/av1/decoder/decoder.c
@@ -58,8 +58,8 @@
mi_params->mi_rows = aligned_height >> MI_SIZE_LOG2;
mi_params->mi_stride = calc_mi_size(mi_params->mi_cols);
- mi_params->mb_cols = (mi_params->mi_cols + 2) >> 2;
- mi_params->mb_rows = (mi_params->mi_rows + 2) >> 2;
+ mi_params->mb_cols = ROUND_POWER_OF_TWO(mi_params->mi_cols, 2);
+ mi_params->mb_rows = ROUND_POWER_OF_TWO(mi_params->mi_rows, 2);
mi_params->MBs = mi_params->mb_rows * mi_params->mb_cols;
mi_params->mi_alloc_bsize = BLOCK_4X4;
diff --git a/av1/ducky_encode.cc b/av1/ducky_encode.cc
index 5bfc124..0b4197b 100644
--- a/av1/ducky_encode.cc
+++ b/av1/ducky_encode.cc
@@ -47,6 +47,7 @@
int g_usage;
int max_ref_frames;
int speed;
+ int base_qindex;
enum aom_rc_mode rc_end_usage;
aom_rational64_t timestamp_ratio;
std::vector<FIRSTPASS_STATS> stats_list;
@@ -54,12 +55,13 @@
};
DuckyEncode::DuckyEncode(const VideoInfo &video_info, int max_ref_frames,
- int speed) {
+ int speed, int base_qindex) {
impl_ptr_ = std::unique_ptr<EncodeImpl>(new EncodeImpl());
impl_ptr_->video_info = video_info;
impl_ptr_->g_usage = GOOD;
impl_ptr_->max_ref_frames = max_ref_frames;
impl_ptr_->speed = speed;
+ impl_ptr_->base_qindex = base_qindex;
impl_ptr_->rc_end_usage = AOM_Q;
// TODO(angiebird): Set timestamp_ratio properly
// timestamp_ratio.den = cfg->g_timebase.den;
@@ -81,6 +83,10 @@
// g_timebase is the inverse of frame_rate
cfg.g_timebase.num = video_info.frame_rate.den;
cfg.g_timebase.den = video_info.frame_rate.num;
+ if (pass == AOM_RC_SECOND_PASS) {
+ cfg.rc_twopass_stats_in.sz =
+ (video_info.frame_count + 1) * sizeof(FIRSTPASS_STATS);
+ }
AV1EncoderConfig oxcf = av1_get_encoder_config(&cfg);
// TODO(angiebird): Why didn't we init use_highbitdepth in
// av1_get_encoder_config()?
@@ -412,8 +418,10 @@
block_stats.col = mi_col * MI_SIZE;
block_stats.height = (1 << block_mis_log2) * MI_SIZE;
block_stats.width = (1 << block_mis_log2) * MI_SIZE;
- block_stats.inter_cost = tpl_stats_ptr->inter_cost;
- block_stats.intra_cost = tpl_stats_ptr->intra_cost;
+ block_stats.inter_cost = tpl_stats_ptr->inter_cost
+ << TPL_DEP_COST_SCALE_LOG2;
+ block_stats.intra_cost = tpl_stats_ptr->intra_cost
+ << TPL_DEP_COST_SCALE_LOG2;
block_stats.ref_frame_index = { -1, -1 };
for (int i = 0; i < kBlockRefCount; ++i) {
@@ -455,7 +463,8 @@
// encoding frame frame_number
aom::EncodeFrameDecision frame_decision = { aom::EncodeFrameMode::kQindex,
aom::EncodeGopMode::kGopRcl,
- { 128, -1 } };
+ { impl_ptr_->base_qindex,
+ -1 } };
(void)frame;
EncodeFrame(frame_decision);
if (ppi->cpi->common.show_frame) pending_ctx_size_ = 0;
diff --git a/av1/ducky_encode.h b/av1/ducky_encode.h
index ffa53b0..086681b 100644
--- a/av1/ducky_encode.h
+++ b/av1/ducky_encode.h
@@ -74,7 +74,7 @@
class DuckyEncode {
public:
explicit DuckyEncode(const VideoInfo &video_info, int max_ref_frames,
- int speed = 3);
+ int speed, int base_qindex);
~DuckyEncode();
std::vector<FIRSTPASS_STATS> ComputeFirstPassStats();
void StartEncode(const std::vector<FIRSTPASS_STATS> &stats_list);
diff --git a/av1/encoder/aq_cyclicrefresh.c b/av1/encoder/aq_cyclicrefresh.c
index 89c4ac3..133e482 100644
--- a/av1/encoder/aq_cyclicrefresh.c
+++ b/av1/encoder/aq_cyclicrefresh.c
@@ -26,6 +26,8 @@
cr->map = aom_calloc(mi_rows * mi_cols, sizeof(*cr->map));
cr->counter_encode_maxq_scene_change = 0;
+ cr->percent_refresh_adjustment = 5;
+ cr->rate_ratio_qdelta_adjustment = 0.25;
if (cr->map == NULL) {
av1_cyclic_refresh_free(cr);
return NULL;
@@ -343,8 +345,7 @@
// Loop through all MI blocks in superblock and update map.
xmis = AOMMIN(mi_params->mi_cols - mi_col, cm->seq_params->mib_size);
ymis = AOMMIN(mi_params->mi_rows - mi_row, cm->seq_params->mib_size);
- if (cpi->sf.rt_sf.sad_based_comp_prune && cr->use_block_sad_scene_det &&
- cpi->rc.frames_since_key > 30 &&
+ if (cr->use_block_sad_scene_det && cpi->rc.frames_since_key > 30 &&
cr->counter_encode_maxq_scene_change > 30 &&
cpi->src_sad_blk_64x64 != NULL) {
sb_sad = cpi->src_sad_blk_64x64[sb_col_index + sb_cols * sb_row_index];
@@ -408,6 +409,14 @@
const int scene_change_detected =
cpi->rc.high_source_sad ||
(cpi->ppi->use_svc && cpi->svc.high_source_sad_superframe);
+
+ // Cases to reset the cyclic refresh adjustment parameters.
+ if (frame_is_intra_only(cm) || scene_change_detected) {
+ // Reset adaptive elements for intra only frames and scene changes.
+ cr->percent_refresh_adjustment = 5;
+ cr->rate_ratio_qdelta_adjustment = 0.25;
+ }
+
// Although this segment feature for RTC is only used for
// blocks >= 8X8, for more efficient coding of the seg map
// cur_frame->seg_map needs to set at 4x4 along with the
@@ -417,6 +426,8 @@
// Also if loop-filter deltas is applied via segment, then
// we need to set cr->skip_over4x4 = 1.
cr->skip_over4x4 = (cpi->oxcf.speed > 9) ? 1 : 0;
+
+ // should we enable cyclic refresh on this frame.
cr->apply_cyclic_refresh = 1;
if (frame_is_intra_only(cm) || is_lossless_requested(&cpi->oxcf.rc_cfg) ||
scene_change_detected || cpi->svc.temporal_layer_id > 0 ||
@@ -430,14 +441,13 @@
cr->apply_cyclic_refresh = 0;
return;
}
- cr->percent_refresh = 10;
- // Increase the amount of refresh for #temporal_layers > 2, and for some
- // frames after scene change that is encoded at high Q.
+
+ // Increase the amount of refresh for #temporal_layers > 2
if (cpi->svc.number_temporal_layers > 2)
cr->percent_refresh = 15;
- else if (cpi->oxcf.tune_cfg.content == AOM_CONTENT_SCREEN &&
- cr->counter_encode_maxq_scene_change < 20)
- cr->percent_refresh = 15;
+ else
+ cr->percent_refresh = 10 + cr->percent_refresh_adjustment;
+
cr->max_qdelta_perc = 60;
cr->time_for_refresh = 0;
cr->use_block_sad_scene_det =
@@ -454,9 +464,9 @@
if (cr->percent_refresh > 0 &&
rc->frames_since_key <
(4 * cpi->svc.number_temporal_layers) * (100 / cr->percent_refresh)) {
- cr->rate_ratio_qdelta = 3.0;
+ cr->rate_ratio_qdelta = 3.0 + cr->rate_ratio_qdelta_adjustment;
} else {
- cr->rate_ratio_qdelta = 2.0;
+ cr->rate_ratio_qdelta = 2.25 + cr->rate_ratio_qdelta_adjustment;
}
// Adjust some parameters for low resolutions.
if (cm->width * cm->height <= 352 * 288) {
@@ -514,6 +524,7 @@
const int resolution_change =
cm->prev_frame && (cm->width != cm->prev_frame->width ||
cm->height != cm->prev_frame->height);
+
if (resolution_change) av1_cyclic_refresh_reset_resize(cpi);
if (!cr->apply_cyclic_refresh) {
// Set segmentation map to 0 and disable.
@@ -601,6 +612,8 @@
cpi->refresh_frame.golden_frame = true;
cr->apply_cyclic_refresh = 0;
cr->counter_encode_maxq_scene_change = 0;
+ cr->percent_refresh_adjustment = 5;
+ cr->rate_ratio_qdelta_adjustment = 0.25;
}
int av1_cyclic_refresh_disable_lf_cdef(AV1_COMP *const cpi) {
diff --git a/av1/encoder/aq_cyclicrefresh.h b/av1/encoder/aq_cyclicrefresh.h
index ecb5d6d..6c2566c 100644
--- a/av1/encoder/aq_cyclicrefresh.h
+++ b/av1/encoder/aq_cyclicrefresh.h
@@ -39,6 +39,12 @@
* for cyclic refresh.
*/
int percent_refresh;
+
+ /*!
+ * Active adjustment delta for cyclic refresh for rate control.
+ */
+ int percent_refresh_adjustment;
+
/*!
* Maximum q-delta as percentage of base q.
*/
@@ -94,6 +100,12 @@
* Rate target ratio to set q delta.
*/
double rate_ratio_qdelta;
+
+ /*!
+ * Active adjustment of qdelta rate ratio for enhanced rate control
+ */
+ double rate_ratio_qdelta_adjustment;
+
/*!
* Boost factor for rate target ratio, for segment CR_SEGMENT_ID_BOOST2.
*/
@@ -172,7 +184,7 @@
* \param[in] mi_col Col coordinate of the block in a step size of MI_SIZE
* \param[in] bsize Block size
*
- * \return Update the \c mbmi->segment_id, the \c cpi->cyclic_refresh and
+ * \remark Update the \c mbmi->segment_id, the \c cpi->cyclic_refresh and
* the \c cm->cpi->enc_seg.map.
*/
@@ -201,7 +213,7 @@
* \param[in] dry_run A code indicating whether it is part of the final
* pass for reconstructing the superblock
*
- * \return Update the \c mbmi->segment_id, the \c cpi->cyclic_refresh and
+ * \remark Update the \c mbmi->segment_id, the \c cpi->cyclic_refresh and
* the \c cm->cpi->enc_seg.map.
*/
void av1_cyclic_refresh_update_segment(const struct AV1_COMP *cpi,
@@ -221,7 +233,7 @@
*
* \param[in] x Pointer to MACROBLOCK structure
*
- * \return Update the \c x->cnt_zeromv, the \c x->actual_num_seg1_blocks and
+ * \remark Update the \c x->cnt_zeromv, the \c x->actual_num_seg1_blocks and
* the \c x->actual_num_seg1_blocks.
*/
void av1_init_cyclic_refresh_counters(MACROBLOCK *const x);
@@ -238,7 +250,7 @@
* \param[in] cyclic_refresh Pointer to CYCLIC_REFRESH structure
* \param[in] x Pointer to MACROBLOCK structure
*
- * \return Update the \c cyclic_refresh->cnt_zeromv, the \c
+ * \remark Update the \c cyclic_refresh->cnt_zeromv, the \c
* cyclic_refresh->actual_num_seg1_blocks and the \c
* cyclic_refresh->actual_num_seg1_blocks.
*/
@@ -253,7 +265,7 @@
*
* \param[in] cpi Top level encoder structure
*
- * \return Returns the interval in \c cpi->rc.baseline_gf_interval.
+ * \remark Returns the interval in \c cpi->rc.baseline_gf_interval.
*/
void av1_cyclic_refresh_set_golden_update(struct AV1_COMP *const cpi);
@@ -270,7 +282,7 @@
*
* \param[in] cpi Top level encoder structure
*
- * \return Updates the \c cpi->cyclic_refresh with the settings.
+ * \remark Updates the \c cpi->cyclic_refresh with the settings.
*/
void av1_cyclic_refresh_update_parameters(struct AV1_COMP *const cpi);
@@ -284,7 +296,7 @@
*
* \param[in] cpi Top level encoder structure
*
- * \return Updates the \c cpi->cyclic_refresh with the cyclic refresh
+ * \remark Updates the \c cpi->cyclic_refresh with the cyclic refresh
* parameters and the \c cm->seg with the segmentation data.
*/
void av1_cyclic_refresh_setup(struct AV1_COMP *const cpi);
diff --git a/av1/encoder/arm/crc32/hash_crc32.c b/av1/encoder/arm/crc32/hash_crc32.c
new file mode 100644
index 0000000..dd8685d
--- /dev/null
+++ b/av1/encoder/arm/crc32/hash_crc32.c
@@ -0,0 +1,54 @@
+/*
+ * Copyright (c) 2022, Alliance for Open Media. All rights reserved
+ *
+ * This source code is subject to the terms of the BSD 2 Clause License and
+ * the Alliance for Open Media Patent License 1.0. If the BSD 2 Clause License
+ * was not distributed with this source code in the LICENSE file, you can
+ * obtain it at www.aomedia.org/license/software. If the Alliance for Open
+ * Media Patent License 1.0 was not distributed with this source code in the
+ * PATENTS file, you can obtain it at www.aomedia.org/license/patent.
+ */
+
+#include <stdint.h>
+#include <stddef.h>
+#include <arm_acle.h>
+
+#define CRC_LOOP(op, crc, type, buf, len) \
+ while ((len) >= sizeof(type)) { \
+ (crc) = op((crc), *(type *)(buf)); \
+ (len) -= sizeof(type); \
+ buf += sizeof(type); \
+ }
+
+#define CRC_SINGLE(op, crc, type, buf, len) \
+ if ((len) >= sizeof(type)) { \
+ (crc) = op((crc), *(type *)(buf)); \
+ (len) -= sizeof(type); \
+ buf += sizeof(type); \
+ }
+
+/* Return 32-bit CRC for the input buffer.
+ * Polynomial is 0x1EDC6F41.
+ */
+
+uint32_t av1_get_crc32c_value_arm_crc32(void *crc_calculator, uint8_t *p,
+ size_t len) {
+ (void)crc_calculator;
+ const uint8_t *buf = p;
+ uint32_t crc = 0xFFFFFFFF;
+
+#if !defined(__aarch64__)
+ // Align input to 8-byte boundary (only necessary for 32-bit builds.)
+ while (len && ((uintptr_t)buf & 7)) {
+ crc = __crc32cb(crc, *buf++);
+ len--;
+ }
+#endif
+
+ CRC_LOOP(__crc32cd, crc, uint64_t, buf, len)
+ CRC_SINGLE(__crc32cw, crc, uint32_t, buf, len)
+ CRC_SINGLE(__crc32ch, crc, uint16_t, buf, len)
+ CRC_SINGLE(__crc32cb, crc, uint8_t, buf, len)
+
+ return ~crc;
+}
diff --git a/av1/encoder/arm/neon/temporal_filter_neon.c b/av1/encoder/arm/neon/temporal_filter_neon.c
new file mode 100644
index 0000000..65fb332b
--- /dev/null
+++ b/av1/encoder/arm/neon/temporal_filter_neon.c
@@ -0,0 +1,432 @@
+/*
+ * Copyright (c) 2022, Alliance for Open Media. All rights reserved
+ *
+ * This source code is subject to the terms of the BSD 2 Clause License and
+ * the Alliance for Open Media Patent License 1.0. If the BSD 2 Clause License
+ * was not distributed with this source code in the LICENSE file, you can
+ * obtain it at www.aomedia.org/license/software. If the Alliance for Open
+ * Media Patent License 1.0 was not distributed with this source code in the
+ * PATENTS file, you can obtain it at www.aomedia.org/license/patent.
+ */
+
+#include <arm_neon.h>
+
+#include "config/av1_rtcd.h"
+#include "av1/encoder/encoder.h"
+#include "av1/encoder/temporal_filter.h"
+#include "aom_dsp/arm/mem_neon.h"
+#include "aom_dsp/arm/sum_neon.h"
+
+// For the squared error buffer, add padding for 4 samples.
+#define SSE_STRIDE (BW + 4)
+
+#if defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+
+// clang-format off
+
+DECLARE_ALIGNED(16, static const uint8_t, kSlidingWindowMask[]) = {
+ 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x00, 0x00, 0x00,
+ 0x00, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x00, 0x00,
+ 0x00, 0x00, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x00,
+ 0x00, 0x00, 0x00, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF
+};
+
+// clang-format on
+
+static INLINE void get_abs_diff(const uint8_t *frame1, const uint32_t stride1,
+ const uint8_t *frame2, const uint32_t stride2,
+ const uint32_t block_width,
+ const uint32_t block_height,
+ uint8_t *frame_abs_diff,
+ const unsigned int dst_stride) {
+ uint8_t *dst = frame_abs_diff;
+
+ uint32_t i = 0;
+ do {
+ uint32_t j = 0;
+ do {
+ uint8x16_t s = vld1q_u8(frame1 + i * stride1 + j);
+ uint8x16_t r = vld1q_u8(frame2 + i * stride2 + j);
+ uint8x16_t abs_diff = vabdq_u8(s, r);
+ vst1q_u8(dst + j + 2, abs_diff);
+ j += 16;
+ } while (j < block_width);
+
+ dst += dst_stride;
+ i++;
+ } while (i < block_height);
+}
+
+static INLINE uint8x16_t load_and_pad(uint8_t *src, const uint32_t col,
+ const uint32_t block_width) {
+ uint8x8_t s = vld1_u8(src);
+
+ if (col == 0) {
+ s[0] = s[2];
+ s[1] = s[2];
+ } else if (col >= block_width - 4) {
+ s[6] = s[5];
+ s[7] = s[5];
+ }
+ return vcombine_u8(s, s);
+}
+
+static void apply_temporal_filter(
+ const uint8_t *frame, const unsigned int stride, const uint32_t block_width,
+ const uint32_t block_height, const int *subblock_mses,
+ unsigned int *accumulator, uint16_t *count, uint8_t *frame_abs_diff,
+ uint32_t *luma_sse_sum, const double inv_num_ref_pixels,
+ const double decay_factor, const double inv_factor,
+ const double weight_factor, double *d_factor) {
+ assert(((block_width == 16) || (block_width == 32)) &&
+ ((block_height == 16) || (block_height == 32)));
+
+ uint32_t acc_5x5_neon[BH][BW];
+ const uint8x16x2_t vmask = vld1q_u8_x2(kSlidingWindowMask);
+
+ // Traverse 4 columns at a time - first and last two columns need padding.
+ for (uint32_t col = 0; col < block_width; col += 4) {
+ uint8x16_t vsrc[5][2];
+ uint8_t *src = frame_abs_diff + col;
+
+ // Load, pad (for first and last two columns) and mask 3 rows from the top.
+ for (int i = 2; i < 5; i++) {
+ uint8x16_t s = load_and_pad(src, col, block_width);
+ vsrc[i][0] = vandq_u8(s, vmask.val[0]);
+ vsrc[i][1] = vandq_u8(s, vmask.val[1]);
+ src += SSE_STRIDE;
+ }
+
+ // Pad the top 2 rows.
+ vsrc[0][0] = vsrc[2][0];
+ vsrc[0][1] = vsrc[2][1];
+ vsrc[1][0] = vsrc[2][0];
+ vsrc[1][1] = vsrc[2][1];
+
+ for (unsigned int row = 0; row < block_height; row++) {
+ uint32x4_t sum_01 = vdupq_n_u32(0);
+ uint32x4_t sum_23 = vdupq_n_u32(0);
+
+ sum_01 = vdotq_u32(sum_01, vsrc[0][0], vsrc[0][0]);
+ sum_01 = vdotq_u32(sum_01, vsrc[1][0], vsrc[1][0]);
+ sum_01 = vdotq_u32(sum_01, vsrc[2][0], vsrc[2][0]);
+ sum_01 = vdotq_u32(sum_01, vsrc[3][0], vsrc[3][0]);
+ sum_01 = vdotq_u32(sum_01, vsrc[4][0], vsrc[4][0]);
+
+ sum_23 = vdotq_u32(sum_23, vsrc[0][1], vsrc[0][1]);
+ sum_23 = vdotq_u32(sum_23, vsrc[1][1], vsrc[1][1]);
+ sum_23 = vdotq_u32(sum_23, vsrc[2][1], vsrc[2][1]);
+ sum_23 = vdotq_u32(sum_23, vsrc[3][1], vsrc[3][1]);
+ sum_23 = vdotq_u32(sum_23, vsrc[4][1], vsrc[4][1]);
+
+ vst1q_u32(&acc_5x5_neon[row][col], vpaddq_u32(sum_01, sum_23));
+
+ // Push all rows in the sliding window up one.
+ for (int i = 0; i < 4; i++) {
+ vsrc[i][0] = vsrc[i + 1][0];
+ vsrc[i][1] = vsrc[i + 1][1];
+ }
+
+ if (row <= block_height - 4) {
+ // Load next row into the bottom of the sliding window.
+ uint8x16_t s = load_and_pad(src, col, block_width);
+ vsrc[4][0] = vandq_u8(s, vmask.val[0]);
+ vsrc[4][1] = vandq_u8(s, vmask.val[1]);
+ src += SSE_STRIDE;
+ } else {
+ // Pad the bottom 2 rows.
+ vsrc[4][0] = vsrc[3][0];
+ vsrc[4][1] = vsrc[3][1];
+ }
+ }
+ }
+
+ // Perform filtering.
+ for (unsigned int i = 0, k = 0; i < block_height; i++) {
+ for (unsigned int j = 0; j < block_width; j++, k++) {
+ const int pixel_value = frame[i * stride + j];
+ uint32_t diff_sse = acc_5x5_neon[i][j] + luma_sse_sum[i * BW + j];
+
+ const double window_error = diff_sse * inv_num_ref_pixels;
+ const int subblock_idx =
+ (i >= block_height / 2) * 2 + (j >= block_width / 2);
+ const double block_error = (double)subblock_mses[subblock_idx];
+ const double combined_error =
+ weight_factor * window_error + block_error * inv_factor;
+ // Compute filter weight.
+ double scaled_error =
+ combined_error * d_factor[subblock_idx] * decay_factor;
+ scaled_error = AOMMIN(scaled_error, 7);
+ const int weight = (int)(exp(-scaled_error) * TF_WEIGHT_SCALE);
+ accumulator[k] += weight * pixel_value;
+ count[k] += weight;
+ }
+ }
+}
+
+#else // !(defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD))
+
+DECLARE_ALIGNED(16, static const uint16_t, kSlidingWindowMask[]) = {
+ 0xFFFF, 0xFFFF, 0xFFFF, 0xFFFF, 0xFFFF, 0x0000, 0x0000, 0x0000,
+ 0x0000, 0xFFFF, 0xFFFF, 0xFFFF, 0xFFFF, 0xFFFF, 0x0000, 0x0000,
+ 0x0000, 0x0000, 0xFFFF, 0xFFFF, 0xFFFF, 0xFFFF, 0xFFFF, 0x0000,
+ 0x0000, 0x0000, 0x0000, 0xFFFF, 0xFFFF, 0xFFFF, 0xFFFF, 0xFFFF
+};
+
+static INLINE void get_squared_error(
+ const uint8_t *frame1, const uint32_t stride1, const uint8_t *frame2,
+ const uint32_t stride2, const uint32_t block_width,
+ const uint32_t block_height, uint16_t *frame_sse,
+ const unsigned int dst_stride) {
+ uint16_t *dst = frame_sse;
+
+ uint32_t i = 0;
+ do {
+ uint32_t j = 0;
+ do {
+ uint8x16_t s = vld1q_u8(frame1 + i * stride1 + j);
+ uint8x16_t r = vld1q_u8(frame2 + i * stride2 + j);
+
+ uint8x16_t abs_diff = vabdq_u8(s, r);
+ uint16x8_t sse_lo =
+ vmull_u8(vget_low_u8(abs_diff), vget_low_u8(abs_diff));
+ uint16x8_t sse_hi =
+ vmull_u8(vget_high_u8(abs_diff), vget_high_u8(abs_diff));
+
+ vst1q_u16(dst + j + 2, sse_lo);
+ vst1q_u16(dst + j + 10, sse_hi);
+
+ j += 16;
+ } while (j < block_width);
+
+ dst += dst_stride;
+ i++;
+ } while (i < block_height);
+}
+
+static INLINE uint16x8_t load_and_pad(uint16_t *src, const uint32_t col,
+ const uint32_t block_width) {
+ uint16x8_t s = vld1q_u16(src);
+
+ if (col == 0) {
+ s[0] = s[2];
+ s[1] = s[2];
+ } else if (col >= block_width - 4) {
+ s[6] = s[5];
+ s[7] = s[5];
+ }
+ return s;
+}
+
+static void apply_temporal_filter(
+ const uint8_t *frame, const unsigned int stride, const uint32_t block_width,
+ const uint32_t block_height, const int *subblock_mses,
+ unsigned int *accumulator, uint16_t *count, uint16_t *frame_sse,
+ uint32_t *luma_sse_sum, const double inv_num_ref_pixels,
+ const double decay_factor, const double inv_factor,
+ const double weight_factor, double *d_factor) {
+ assert(((block_width == 16) || (block_width == 32)) &&
+ ((block_height == 16) || (block_height == 32)));
+
+ uint32_t acc_5x5_neon[BH][BW];
+ const uint16x8x4_t vmask = vld1q_u16_x4(kSlidingWindowMask);
+
+ // Traverse 4 columns at a time - first and last two columns need padding.
+ for (uint32_t col = 0; col < block_width; col += 4) {
+ uint16x8_t vsrc[5];
+ uint16_t *src = frame_sse + col;
+
+ // Load and pad (for first and last two columns) 3 rows from the top.
+ for (int i = 2; i < 5; i++) {
+ vsrc[i] = load_and_pad(src, col, block_width);
+ src += SSE_STRIDE;
+ }
+
+ // Pad the top 2 rows.
+ vsrc[0] = vsrc[2];
+ vsrc[1] = vsrc[2];
+
+ for (unsigned int row = 0; row < block_height; row++) {
+ for (int i = 0; i < 4; i++) {
+ uint32x4_t vsum = vdupq_n_u32(0);
+ for (int j = 0; j < 5; j++) {
+ vsum = vpadalq_u16(vsum, vandq_u16(vsrc[j], vmask.val[i]));
+ }
+ acc_5x5_neon[row][col + i] = horizontal_add_u32x4(vsum);
+ }
+
+ // Push all rows in the sliding window up one.
+ for (int i = 0; i < 4; i++) {
+ vsrc[i] = vsrc[i + 1];
+ }
+
+ if (row <= block_height - 4) {
+ // Load next row into the bottom of the sliding window.
+ vsrc[4] = load_and_pad(src, col, block_width);
+ src += SSE_STRIDE;
+ } else {
+ // Pad the bottom 2 rows.
+ vsrc[4] = vsrc[3];
+ }
+ }
+ }
+
+ // Perform filtering.
+ for (unsigned int i = 0, k = 0; i < block_height; i++) {
+ for (unsigned int j = 0; j < block_width; j++, k++) {
+ const int pixel_value = frame[i * stride + j];
+ uint32_t diff_sse = acc_5x5_neon[i][j] + luma_sse_sum[i * BW + j];
+
+ const double window_error = diff_sse * inv_num_ref_pixels;
+ const int subblock_idx =
+ (i >= block_height / 2) * 2 + (j >= block_width / 2);
+ const double block_error = (double)subblock_mses[subblock_idx];
+ const double combined_error =
+ weight_factor * window_error + block_error * inv_factor;
+ // Compute filter weight.
+ double scaled_error =
+ combined_error * d_factor[subblock_idx] * decay_factor;
+ scaled_error = AOMMIN(scaled_error, 7);
+ const int weight = (int)(exp(-scaled_error) * TF_WEIGHT_SCALE);
+ accumulator[k] += weight * pixel_value;
+ count[k] += weight;
+ }
+ }
+}
+
+#endif // defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+
+void av1_apply_temporal_filter_neon(
+ const YV12_BUFFER_CONFIG *frame_to_filter, const MACROBLOCKD *mbd,
+ const BLOCK_SIZE block_size, const int mb_row, const int mb_col,
+ const int num_planes, const double *noise_levels, const MV *subblock_mvs,
+ const int *subblock_mses, const int q_factor, const int filter_strength,
+ const uint8_t *pred, uint32_t *accum, uint16_t *count) {
+ const int is_high_bitdepth = frame_to_filter->flags & YV12_FLAG_HIGHBITDEPTH;
+ assert(block_size == BLOCK_32X32 && "Only support 32x32 block with Neon!");
+ assert(TF_WINDOW_LENGTH == 5 && "Only support window length 5 with Neon!");
+ assert(!is_high_bitdepth && "Only support low bit-depth with Neon!");
+ assert(num_planes >= 1 && num_planes <= MAX_MB_PLANE);
+ (void)is_high_bitdepth;
+
+ // Block information.
+ const int mb_height = block_size_high[block_size];
+ const int mb_width = block_size_wide[block_size];
+ // Frame information.
+ const int frame_height = frame_to_filter->y_crop_height;
+ const int frame_width = frame_to_filter->y_crop_width;
+ const int min_frame_size = AOMMIN(frame_height, frame_width);
+ // Variables to simplify combined error calculation.
+ const double inv_factor = 1.0 / ((TF_WINDOW_BLOCK_BALANCE_WEIGHT + 1) *
+ TF_SEARCH_ERROR_NORM_WEIGHT);
+ const double weight_factor =
+ (double)TF_WINDOW_BLOCK_BALANCE_WEIGHT * inv_factor;
+ // Adjust filtering based on q.
+ // Larger q -> stronger filtering -> larger weight.
+ // Smaller q -> weaker filtering -> smaller weight.
+ double q_decay = pow((double)q_factor / TF_Q_DECAY_THRESHOLD, 2);
+ q_decay = CLIP(q_decay, 1e-5, 1);
+ if (q_factor >= TF_QINDEX_CUTOFF) {
+ // Max q_factor is 255, therefore the upper bound of q_decay is 8.
+ // We do not need a clip here.
+ q_decay = 0.5 * pow((double)q_factor / 64, 2);
+ }
+ // Smaller strength -> smaller filtering weight.
+ double s_decay = pow((double)filter_strength / TF_STRENGTH_THRESHOLD, 2);
+ s_decay = CLIP(s_decay, 1e-5, 1);
+ double d_factor[4] = { 0 };
+#if defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+ uint8_t frame_abs_diff[SSE_STRIDE * BH] = { 0 };
+#else // !(defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD))
+ uint16_t frame_sse[SSE_STRIDE * BH] = { 0 };
+#endif // defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+ uint32_t luma_sse_sum[BW * BH] = { 0 };
+
+ for (int subblock_idx = 0; subblock_idx < 4; subblock_idx++) {
+ // Larger motion vector -> smaller filtering weight.
+ const MV mv = subblock_mvs[subblock_idx];
+ const double distance = sqrt(pow(mv.row, 2) + pow(mv.col, 2));
+ double distance_threshold = min_frame_size * TF_SEARCH_DISTANCE_THRESHOLD;
+ distance_threshold = AOMMAX(distance_threshold, 1);
+ d_factor[subblock_idx] = distance / distance_threshold;
+ d_factor[subblock_idx] = AOMMAX(d_factor[subblock_idx], 1);
+ }
+
+ // Handle planes in sequence.
+ int plane_offset = 0;
+ for (int plane = 0; plane < num_planes; ++plane) {
+ const uint32_t plane_h = mb_height >> mbd->plane[plane].subsampling_y;
+ const uint32_t plane_w = mb_width >> mbd->plane[plane].subsampling_x;
+ const uint32_t frame_stride =
+ frame_to_filter->strides[plane == AOM_PLANE_Y ? 0 : 1];
+ const int frame_offset = mb_row * plane_h * frame_stride + mb_col * plane_w;
+
+ const uint8_t *ref = frame_to_filter->buffers[plane] + frame_offset;
+ const int ss_x_shift =
+ mbd->plane[plane].subsampling_x - mbd->plane[AOM_PLANE_Y].subsampling_x;
+ const int ss_y_shift =
+ mbd->plane[plane].subsampling_y - mbd->plane[AOM_PLANE_Y].subsampling_y;
+ const int num_ref_pixels = TF_WINDOW_LENGTH * TF_WINDOW_LENGTH +
+ ((plane) ? (1 << (ss_x_shift + ss_y_shift)) : 0);
+ const double inv_num_ref_pixels = 1.0 / num_ref_pixels;
+ // Larger noise -> larger filtering weight.
+ const double n_decay = 0.5 + log(2 * noise_levels[plane] + 5.0);
+ // Decay factors for non-local mean approach.
+ const double decay_factor = 1 / (n_decay * q_decay * s_decay);
+
+ // Filter U-plane and V-plane using Y-plane. This is because motion
+ // search is only done on Y-plane, so the information from Y-plane
+ // will be more accurate. The luma sse sum is reused in both chroma
+ // planes.
+#if defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+ if (plane == AOM_PLANE_U) {
+ for (unsigned int i = 0; i < plane_h; i++) {
+ for (unsigned int j = 0; j < plane_w; j++) {
+ for (int ii = 0; ii < (1 << ss_y_shift); ++ii) {
+ for (int jj = 0; jj < (1 << ss_x_shift); ++jj) {
+ const int yy = (i << ss_y_shift) + ii; // Y-coord on Y-plane.
+ const int xx = (j << ss_x_shift) + jj; // X-coord on Y-plane.
+ luma_sse_sum[i * BW + j] +=
+ (frame_abs_diff[yy * SSE_STRIDE + xx + 2] *
+ frame_abs_diff[yy * SSE_STRIDE + xx + 2]);
+ }
+ }
+ }
+ }
+ }
+
+ get_abs_diff(ref, frame_stride, pred + plane_offset, plane_w, plane_w,
+ plane_h, frame_abs_diff, SSE_STRIDE);
+
+ apply_temporal_filter(pred + plane_offset, plane_w, plane_w, plane_h,
+ subblock_mses, accum + plane_offset,
+ count + plane_offset, frame_abs_diff, luma_sse_sum,
+ inv_num_ref_pixels, decay_factor, inv_factor,
+ weight_factor, d_factor);
+#else // !(defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD))
+ if (plane == AOM_PLANE_U) {
+ for (unsigned int i = 0; i < plane_h; i++) {
+ for (unsigned int j = 0; j < plane_w; j++) {
+ for (int ii = 0; ii < (1 << ss_y_shift); ++ii) {
+ for (int jj = 0; jj < (1 << ss_x_shift); ++jj) {
+ const int yy = (i << ss_y_shift) + ii; // Y-coord on Y-plane.
+ const int xx = (j << ss_x_shift) + jj; // X-coord on Y-plane.
+ luma_sse_sum[i * BW + j] += frame_sse[yy * SSE_STRIDE + xx + 2];
+ }
+ }
+ }
+ }
+ }
+
+ get_squared_error(ref, frame_stride, pred + plane_offset, plane_w, plane_w,
+ plane_h, frame_sse, SSE_STRIDE);
+
+ apply_temporal_filter(
+ pred + plane_offset, plane_w, plane_w, plane_h, subblock_mses,
+ accum + plane_offset, count + plane_offset, frame_sse, luma_sse_sum,
+ inv_num_ref_pixels, decay_factor, inv_factor, weight_factor, d_factor);
+#endif // defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD)
+
+ plane_offset += plane_h * plane_w;
+ }
+}
diff --git a/av1/encoder/arm/neon/wedge_utils_neon.c b/av1/encoder/arm/neon/wedge_utils_neon.c
new file mode 100644
index 0000000..54d8d19
--- /dev/null
+++ b/av1/encoder/arm/neon/wedge_utils_neon.c
@@ -0,0 +1,77 @@
+/*
+ * Copyright (c) 2022, Alliance for Open Media. All rights reserved
+ *
+ * This source code is subject to the terms of the BSD 2 Clause License and
+ * the Alliance for Open Media Patent License 1.0. If the BSD 2 Clause License
+ * was not distributed with this source code in the LICENSE file, you can
+ * obtain it at www.aomedia.org/license/software. If the Alliance for Open
+ * Media Patent License 1.0 was not distributed with this source code in the
+ * PATENTS file, you can obtain it at www.aomedia.org/license/patent.
+ */
+
+#include <arm_neon.h>
+#include <assert.h>
+
+#include "aom_dsp/arm/sum_neon.h"
+#include "av1/common/reconinter.h"
+
+#define MAX_MASK_VALUE (1 << WEDGE_WEIGHT_BITS)
+
+/**
+ * See av1_wedge_sse_from_residuals_c for details of the parameters and
+ * computation.
+ */
+uint64_t av1_wedge_sse_from_residuals_neon(const int16_t *r1, const int16_t *d,
+ const uint8_t *m, int N) {
+ assert(N % 64 == 0);
+
+ uint64x2_t v_csse[2] = { vdupq_n_u64(0), vdupq_n_u64(0) };
+
+ int i = 0;
+ do {
+ int32x4_t sum[4];
+ int32x4_t sse[2];
+ int16x4_t sum_s16[4];
+
+ const int16x8_t r1_l = vld1q_s16(r1 + i);
+ const int16x8_t r1_h = vld1q_s16(r1 + i + 8);
+ const int16x8_t d_l = vld1q_s16(d + i);
+ const int16x8_t d_h = vld1q_s16(d + i + 8);
+ // The following three lines are a bit inelegant compared to using a pair
+ // of vmovl_u8()... but it forces the compiler to emit a ZIP1, ZIP2 pair -
+ // which can be executed in parallel with the subsequent SSHL instructions.
+ // (SSHL can only be executed on half of the Neon pipes in modern Arm
+ // cores, whereas ZIP1/2 can be executed on all of them.)
+ const uint8x16x2_t m_u16 = vzipq_u8(vld1q_u8(m + i), vdupq_n_u8(0));
+ const int16x8_t m_l = vreinterpretq_s16_u8(m_u16.val[0]);
+ const int16x8_t m_h = vreinterpretq_s16_u8(m_u16.val[1]);
+
+ sum[0] = vshll_n_s16(vget_low_s16(r1_l), WEDGE_WEIGHT_BITS);
+ sum[1] = vshll_n_s16(vget_high_s16(r1_l), WEDGE_WEIGHT_BITS);
+ sum[2] = vshll_n_s16(vget_low_s16(r1_h), WEDGE_WEIGHT_BITS);
+ sum[3] = vshll_n_s16(vget_high_s16(r1_h), WEDGE_WEIGHT_BITS);
+
+ sum[0] = vmlal_s16(sum[0], vget_low_s16(m_l), vget_low_s16(d_l));
+ sum[1] = vmlal_s16(sum[1], vget_high_s16(m_l), vget_high_s16(d_l));
+ sum[2] = vmlal_s16(sum[2], vget_low_s16(m_h), vget_low_s16(d_h));
+ sum[3] = vmlal_s16(sum[3], vget_high_s16(m_h), vget_high_s16(d_h));
+
+ sum_s16[0] = vqmovn_s32(sum[0]);
+ sum_s16[1] = vqmovn_s32(sum[1]);
+ sum_s16[2] = vqmovn_s32(sum[2]);
+ sum_s16[3] = vqmovn_s32(sum[3]);
+
+ sse[0] = vmull_s16(sum_s16[0], sum_s16[0]);
+ sse[1] = vmull_s16(sum_s16[2], sum_s16[2]);
+ sse[0] = vmlal_s16(sse[0], sum_s16[1], sum_s16[1]);
+ sse[1] = vmlal_s16(sse[1], sum_s16[3], sum_s16[3]);
+
+ v_csse[0] = vpadalq_u32(v_csse[0], vreinterpretq_u32_s32(sse[0]));
+ v_csse[1] = vpadalq_u32(v_csse[1], vreinterpretq_u32_s32(sse[1]));
+
+ i += 16;
+ } while (i < N);
+
+ uint64_t csse = horizontal_add_u64x2(vaddq_u64(v_csse[0], v_csse[1]));
+ return ROUND_POWER_OF_TWO(csse, 2 * WEDGE_WEIGHT_BITS);
+}
diff --git a/av1/encoder/av1_temporal_denoiser.c b/av1/encoder/av1_temporal_denoiser.c
index 27a12cb..881d252 100644
--- a/av1/encoder/av1_temporal_denoiser.c
+++ b/av1/encoder/av1_temporal_denoiser.c
@@ -233,7 +233,7 @@
frame == ALTREF_FRAME ||
(frame == GOLDEN_FRAME && use_gf_temporal_ref) ||
(frame != LAST_FRAME &&
- ((ctx->zeromv_lastref_sse<(5 * ctx->zeromv_sse)>> 2) ||
+ ((ctx->zeromv_lastref_sse < (5 * ctx->zeromv_sse) >> 2) ||
denoiser->denoising_level >= kDenHigh))) {
frame = LAST_FRAME;
ctx->newmv_sse = ctx->zeromv_lastref_sse;
@@ -348,7 +348,7 @@
decision = perform_motion_compensation(
&cpi->common, denoiser, mb, bs, increase_denoising, mi_row, mi_col, ctx,
motion_magnitude, &zeromv_filter, cpi->svc.number_spatial_layers,
- cpi->source->y_width, cpi->svc.ref_idx[0], cpi->svc.ref_idx[3],
+ cpi->source->y_width, cpi->rtc_ref.ref_idx[0], cpi->rtc_ref.ref_idx[3],
cpi->ppi->use_svc, cpi->svc.spatial_layer_id, use_gf_temporal_ref);
if (decision == FILTER_BLOCK) {
@@ -395,10 +395,11 @@
}
void av1_denoiser_update_frame_info(
- AV1_DENOISER *denoiser, YV12_BUFFER_CONFIG src, struct SVC *svc,
- FRAME_TYPE frame_type, int refresh_alt_ref_frame, int refresh_golden_frame,
- int refresh_last_frame, int alt_fb_idx, int gld_fb_idx, int lst_fb_idx,
- int resized, int svc_refresh_denoiser_buffers, int second_spatial_layer) {
+ AV1_DENOISER *denoiser, YV12_BUFFER_CONFIG src, struct RTC_REF *rtc_ref,
+ struct SVC *svc, FRAME_TYPE frame_type, int refresh_alt_ref_frame,
+ int refresh_golden_frame, int refresh_last_frame, int alt_fb_idx,
+ int gld_fb_idx, int lst_fb_idx, int resized,
+ int svc_refresh_denoiser_buffers, int second_spatial_layer) {
const int shift = second_spatial_layer ? denoiser->num_ref_frames : 0;
// Copy source into denoised reference buffers on KEY_FRAME or
// if the just encoded frame was resized. For SVC, copy source if the base
@@ -415,10 +416,10 @@
return;
}
- if (svc->set_ref_frame_config) {
+ if (rtc_ref->set_ref_frame_config) {
int i;
for (i = 0; i < REF_FRAMES; i++) {
- if (svc->refresh[svc->spatial_layer_id] & (1 << i))
+ if (rtc_ref->refresh[svc->spatial_layer_id] & (1 << i))
copy_frame(&denoiser->running_avg_y[i + 1 + shift],
&denoiser->running_avg_y[INTRA_FRAME + shift]);
}
@@ -497,15 +498,16 @@
}
int av1_denoiser_realloc_svc(AV1_COMMON *cm, AV1_DENOISER *denoiser,
- struct SVC *svc, int svc_buf_shift,
- int refresh_alt, int refresh_gld, int refresh_lst,
- int alt_fb_idx, int gld_fb_idx, int lst_fb_idx) {
+ struct RTC_REF *rtc_ref, struct SVC *svc,
+ int svc_buf_shift, int refresh_alt,
+ int refresh_gld, int refresh_lst, int alt_fb_idx,
+ int gld_fb_idx, int lst_fb_idx) {
int fail = 0;
- if (svc->set_ref_frame_config) {
+ if (rtc_ref->set_ref_frame_config) {
int i;
for (i = 0; i < REF_FRAMES; i++) {
if (cm->current_frame.frame_type == KEY_FRAME ||
- svc->refresh[svc->spatial_layer_id] & (1 << i)) {
+ rtc_ref->refresh[svc->spatial_layer_id] & (1 << i)) {
fail = av1_denoiser_realloc_svc_helper(cm, denoiser,
i + 1 + svc_buf_shift);
}
@@ -671,7 +673,7 @@
int64_t av1_scale_part_thresh(int64_t threshold, AV1_DENOISER_LEVEL noise_level,
CONTENT_STATE_SB content_state,
int temporal_layer_id) {
- if ((content_state.source_sad_nonrd == kLowSad &&
+ if ((content_state.source_sad_nonrd <= kLowSad &&
content_state.low_sumdiff) ||
(content_state.source_sad_nonrd == kHighSad &&
content_state.low_sumdiff) ||
@@ -691,10 +693,10 @@
AV1_DENOISER_LEVEL noise_level, int abs_sumdiff,
int temporal_layer_id) {
if (noise_level >= kDenLow && abs_sumdiff < 5)
- return threshold *=
- (noise_level == kDenLow) ? 2 : (temporal_layer_id == 2) ? 10 : 6;
- else
- return threshold;
+ threshold *= (noise_level == kDenLow) ? 2
+ : (temporal_layer_id == 2) ? 10
+ : 6;
+ return threshold;
}
void av1_denoiser_reset_on_first_frame(AV1_COMP *const cpi) {
@@ -710,6 +712,7 @@
void av1_denoiser_update_ref_frame(AV1_COMP *const cpi) {
AV1_COMMON *const cm = &cpi->common;
+ RTC_REF *const rtc_ref = &cpi->rtc_ref;
SVC *const svc = &cpi->svc;
if (cpi->oxcf.noise_sensitivity > 0 && denoise_svc(cpi) &&
@@ -739,7 +742,8 @@
svc->number_spatial_layers - svc->spatial_layer_id == 2 ? 1 : 0;
// Check if we need to allocate extra buffers in the denoiser
// for refreshed frames.
- if (av1_denoiser_realloc_svc(cm, &cpi->denoiser, svc, svc_buf_shift,
+ if (av1_denoiser_realloc_svc(cm, &cpi->denoiser, rtc_ref,
+ svc, svc_buf_shift,
cpi->refresh_alt_ref_frame,
cpi->refresh_golden_frame,
cpi->refresh_last_frame, cpi->alt_fb_idx,
@@ -749,10 +753,10 @@
#endif
}
av1_denoiser_update_frame_info(
- &cpi->denoiser, *cpi->source, svc, frame_type,
+ &cpi->denoiser, *cpi->source, rtc_ref, svc, frame_type,
cpi->refresh_frame.alt_ref_frame, cpi->refresh_frame.golden_frame, 1,
- svc->ref_idx[6], svc->ref_idx[3], svc->ref_idx[0], resize_pending,
- svc_refresh_denoiser_buffers, denoise_svc_second_layer);
+ rtc_ref->ref_idx[6], rtc_ref->ref_idx[3], rtc_ref->ref_idx[0],
+ resize_pending, svc_refresh_denoiser_buffers, denoise_svc_second_layer);
}
}
diff --git a/av1/encoder/av1_temporal_denoiser.h b/av1/encoder/av1_temporal_denoiser.h
index 71c8c1c..14dcccc 100644
--- a/av1/encoder/av1_temporal_denoiser.h
+++ b/av1/encoder/av1_temporal_denoiser.h
@@ -69,12 +69,14 @@
struct AV1_COMP;
struct SVC;
+struct RTC_REF;
void av1_denoiser_update_frame_info(
- AV1_DENOISER *denoiser, YV12_BUFFER_CONFIG src, struct SVC *svc,
- FRAME_TYPE frame_type, int refresh_alt_ref_frame, int refresh_golden_frame,
- int refresh_last_frame, int alt_fb_idx, int gld_fb_idx, int lst_fb_idx,
- int resized, int svc_refresh_denoiser_buffers, int second_spatial_layer);
+ AV1_DENOISER *denoiser, YV12_BUFFER_CONFIG src, struct RTC_REF *rtc_ref,
+ struct SVC *svc, FRAME_TYPE frame_type, int refresh_alt_ref_frame,
+ int refresh_golden_frame, int refresh_last_frame, int alt_fb_idx,
+ int gld_fb_idx, int lst_fb_idx, int resized,
+ int svc_refresh_denoiser_buffers, int second_spatial_layer);
void av1_denoiser_denoise(struct AV1_COMP *cpi, MACROBLOCK *mb, int mi_row,
int mi_col, BLOCK_SIZE bs, PICK_MODE_CONTEXT *ctx,
@@ -88,9 +90,10 @@
PICK_MODE_CONTEXT *ctx);
int av1_denoiser_realloc_svc(AV1_COMMON *cm, AV1_DENOISER *denoiser,
- struct SVC *svc, int svc_buf_shift,
- int refresh_alt, int refresh_gld, int refresh_lst,
- int alt_fb_idx, int gld_fb_idx, int lst_fb_idx);
+ struct RTC_REF *rtc, struct SVC *svc,
+ int svc_buf_shift, int refresh_alt,
+ int refresh_gld, int refresh_lst, int alt_fb_idx,
+ int gld_fb_idx, int lst_fb_idx);
int av1_denoiser_alloc(AV1_COMMON *cm, struct SVC *svc, AV1_DENOISER *denoiser,
int use_svc, int noise_sen, int width, int height,
diff --git a/av1/encoder/bitstream.c b/av1/encoder/bitstream.c
index 6e4af81..3f159c0 100644
--- a/av1/encoder/bitstream.c
+++ b/av1/encoder/bitstream.c
@@ -2764,7 +2764,31 @@
}
}
-static int check_frame_refs_short_signaling(AV1_COMMON *const cm) {
+static int check_frame_refs_short_signaling(AV1_COMMON *const cm,
+ bool enable_ref_short_signaling) {
+ // In rtc case when res < 360p and speed >= 9, we turn on
+ // frame_refs_short_signaling if it won't break the decoder.
+ if (enable_ref_short_signaling) {
+ const int gld_map_idx = get_ref_frame_map_idx(cm, GOLDEN_FRAME);
+ const int base =
+ 1 << (cm->seq_params->order_hint_info.order_hint_bits_minus_1 + 1);
+
+ const int order_hint_group_cur =
+ cm->current_frame.display_order_hint / base;
+ const int order_hint_group_gld =
+ cm->ref_frame_map[gld_map_idx]->display_order_hint / base;
+ const int relative_dist = cm->current_frame.order_hint -
+ cm->ref_frame_map[gld_map_idx]->order_hint;
+
+ // If current frame and GOLDEN frame are in the same order_hint group, and
+ // they are not far apart (i.e., > 64 frames), then return 1.
+ if (order_hint_group_cur == order_hint_group_gld && relative_dist >= 0 &&
+ relative_dist <= 64) {
+ return 1;
+ }
+ return 0;
+ }
+
// Check whether all references are distinct frames.
const RefCntBuffer *seen_bufs[FRAME_BUFFERS] = { NULL };
int num_refs = 0;
@@ -2842,7 +2866,13 @@
CurrentFrame *const current_frame = &cm->current_frame;
FeatureFlags *const features = &cm->features;
- current_frame->frame_refs_short_signaling = 0;
+ if (!cpi->sf.rt_sf.enable_ref_short_signaling ||
+ !seq_params->order_hint_info.enable_order_hint ||
+ seq_params->order_hint_info.enable_ref_frame_mvs) {
+ current_frame->frame_refs_short_signaling = 0;
+ } else {
+ current_frame->frame_refs_short_signaling = 1;
+ }
if (seq_params->still_picture) {
assert(cm->show_existing_frame == 0);
@@ -3008,12 +3038,20 @@
#endif // FRAME_REFS_SHORT_SIGNALING
if (current_frame->frame_refs_short_signaling) {
- // NOTE(zoeliu@google.com):
- // An example solution for encoder-side implementation on frame refs
- // short signaling, which is only turned on when the encoder side
- // decision on ref frames is identical to that at the decoder side.
+ // In rtc case when cpi->sf.rt_sf.enable_ref_short_signaling is true,
+ // we turn on frame_refs_short_signaling when the current frame and
+ // golden frame are in the same order_hint group, and their relative
+ // distance is <= 64 (in order to be decodable).
+
+ // For other cases, an example solution for encoder-side
+ // implementation on frame_refs_short_signaling is also provided in
+ // this function, where frame_refs_short_signaling is only turned on
+ // when the encoder side decision on ref frames is identical to that
+ // at the decoder side.
+
current_frame->frame_refs_short_signaling =
- check_frame_refs_short_signaling(cm);
+ check_frame_refs_short_signaling(
+ cm, cpi->sf.rt_sf.enable_ref_short_signaling);
}
if (seq_params->order_hint_info.enable_order_hint)
@@ -3602,7 +3640,7 @@
}
}
- mem_put_le32(buf->data, tile_header);
+ mem_put_le32(buf->data, (MEM_VALUE_T)tile_header);
}
*total_size += tile_size;
diff --git a/av1/encoder/block.h b/av1/encoder/block.h
index 0ad118d..7a35256 100644
--- a/av1/encoder/block.h
+++ b/av1/encoder/block.h
@@ -496,7 +496,7 @@
*/
typedef struct {
//! Whether to skip transform and quantization on a partition block level.
- int skip_txfm;
+ uint8_t skip_txfm;
/*! \brief Whether to skip transform and quantization on a txfm block level.
*
@@ -801,13 +801,14 @@
/*!\cond */
typedef enum {
kZeroSad = 0,
- kLowSad = 1,
- kMedSad = 2,
- kHighSad = 3
+ kVeryLowSad = 1,
+ kLowSad = 2,
+ kMedSad = 3,
+ kHighSad = 4
} SOURCE_SAD;
typedef struct {
- //! SAD levels in non-rd path for var-based part and inter-mode search
+ //! SAD levels in non-rd path
SOURCE_SAD source_sad_nonrd;
//! SAD levels in rd-path for var-based part qindex thresholds
SOURCE_SAD source_sad_rd;
@@ -1016,9 +1017,16 @@
*/
int cnt_zeromv;
- /*!\brief Flag to force zeromv-skip block, for nonrd path.
+ /*!\brief Flag to force zeromv-skip at superblock level, for nonrd path.
+ *
+ * 0/1 imply zeromv-skip is disabled/enabled. 2 implies that the blocks
+ * in the superblock may be marked as zeromv-skip at block level.
*/
- int force_zeromv_skip;
+ int force_zeromv_skip_for_sb;
+
+ /*!\brief Flag to force zeromv-skip at block level, for nonrd path.
+ */
+ int force_zeromv_skip_for_blk;
/*! \brief Previous segment id for which qmatrices were updated.
* This is used to bypass setting of qmatrices if no change in qindex.
diff --git a/av1/encoder/compound_type.c b/av1/encoder/compound_type.c
index 4f762b9..39c505d 100644
--- a/av1/encoder/compound_type.c
+++ b/av1/encoder/compound_type.c
@@ -465,7 +465,8 @@
INTERINTRA_MODE *best_interintra_mode, int64_t *best_interintra_rd,
INTERINTRA_MODE interintra_mode, BLOCK_SIZE bsize) {
const AV1_COMMON *const cm = &cpi->common;
- int rate, skip_txfm_sb;
+ int rate;
+ uint8_t skip_txfm_sb;
int64_t dist, skip_sse_sb;
const int bw = block_size_wide[bsize];
mbmi->interintra_mode = interintra_mode;
@@ -688,7 +689,8 @@
const int_mv mv0 = mbmi->mv[0];
// Refine motion vector for NEWMV case.
if (have_newmv_in_inter_mode(mbmi->mode)) {
- int rate_sum, skip_txfm_sb;
+ int rate_sum;
+ uint8_t skip_txfm_sb;
int64_t dist_sum, skip_sse_sb;
// get negative of mask
const uint8_t *mask =
@@ -1048,7 +1050,8 @@
const COMPOUND_TYPE compound_type = mbmi->interinter_comp.type;
// This function will be called only for COMPOUND_WEDGE and COMPOUND_DIFFWTD
assert(compound_type == COMPOUND_WEDGE || compound_type == COMPOUND_DIFFWTD);
- int rate_sum, tmp_skip_txfm_sb;
+ int rate_sum;
+ uint8_t tmp_skip_txfm_sb;
int64_t dist_sum, tmp_skip_sse_sb;
pick_interinter_mask_type pick_interinter_mask[2] = { pick_interinter_wedge,
pick_interinter_seg };
@@ -1300,7 +1303,8 @@
// Case COMPOUND_AVERAGE and COMPOUND_DISTWTD
if (cur_type < COMPOUND_WEDGE) {
if (cpi->sf.inter_sf.enable_fast_compound_mode_search == 2) {
- int rate_sum, tmp_skip_txfm_sb;
+ int rate_sum;
+ uint8_t tmp_skip_txfm_sb;
int64_t dist_sum, tmp_skip_sse_sb;
// Reuse data if matching record is found
diff --git a/av1/encoder/encode_strategy.c b/av1/encoder/encode_strategy.c
index 76d136c..c6656a5 100644
--- a/av1/encoder/encode_strategy.c
+++ b/av1/encoder/encode_strategy.c
@@ -310,8 +310,7 @@
// Return the frame source, or NULL if we couldn't find one
static struct lookahead_entry *choose_frame_source(
AV1_COMP *const cpi, int *const flush, int *pop_lookahead,
- struct lookahead_entry **last_source,
- EncodeFrameParams *const frame_params) {
+ struct lookahead_entry **last_source, int *const show_frame) {
AV1_COMMON *const cm = &cpi->common;
const GF_GROUP *const gf_group = &cpi->ppi->gf_group;
struct lookahead_entry *source = NULL;
@@ -353,7 +352,7 @@
src_index = 0;
}
- frame_params->show_frame = *pop_lookahead;
+ *show_frame = *pop_lookahead;
#if CONFIG_FPMT_TEST
if (cpi->ppi->fpmt_unit_test_cfg == PARALLEL_ENCODE) {
@@ -365,7 +364,7 @@
!is_stat_generation_stage(cpi))
src_index = gf_group->src_offset[cpi->gf_frame_index];
}
- if (frame_params->show_frame) {
+ if (*show_frame) {
// show frame, pop from buffer
// Get last frame source.
if (cm->current_frame.frame_number > 0) {
@@ -413,35 +412,35 @@
const RefreshFrameInfo *const refresh_frame,
unsigned int *frame_flags) {
if (encode_show_existing_frame(cm)) {
- *frame_flags &= ~FRAMEFLAGS_GOLDEN;
- *frame_flags &= ~FRAMEFLAGS_BWDREF;
- *frame_flags &= ~FRAMEFLAGS_ALTREF;
- *frame_flags &= ~FRAMEFLAGS_KEY;
+ *frame_flags &= ~(uint32_t)FRAMEFLAGS_GOLDEN;
+ *frame_flags &= ~(uint32_t)FRAMEFLAGS_BWDREF;
+ *frame_flags &= ~(uint32_t)FRAMEFLAGS_ALTREF;
+ *frame_flags &= ~(uint32_t)FRAMEFLAGS_KEY;
return;
}
if (refresh_frame->golden_frame) {
*frame_flags |= FRAMEFLAGS_GOLDEN;
} else {
- *frame_flags &= ~FRAMEFLAGS_GOLDEN;
+ *frame_flags &= ~(uint32_t)FRAMEFLAGS_GOLDEN;
}
if (refresh_frame->alt_ref_frame) {
*frame_flags |= FRAMEFLAGS_ALTREF;
} else {
- *frame_flags &= ~FRAMEFLAGS_ALTREF;
+ *frame_flags &= ~(uint32_t)FRAMEFLAGS_ALTREF;
}
if (refresh_frame->bwd_ref_frame) {
*frame_flags |= FRAMEFLAGS_BWDREF;
} else {
- *frame_flags &= ~FRAMEFLAGS_BWDREF;
+ *frame_flags &= ~(uint32_t)FRAMEFLAGS_BWDREF;
}
if (cm->current_frame.frame_type == KEY_FRAME) {
*frame_flags |= FRAMEFLAGS_KEY;
} else {
- *frame_flags &= ~FRAMEFLAGS_KEY;
+ *frame_flags &= ~(uint32_t)FRAMEFLAGS_KEY;
}
}
@@ -616,8 +615,8 @@
// flags to 0 to keep things consistent.
if (frame_params->show_existing_frame) return 0;
- const SVC *const svc = &cpi->svc;
- if (is_frame_droppable(svc, ext_refresh_frame_flags)) return 0;
+ const RTC_REF *const rtc_ref = &cpi->rtc_ref;
+ if (is_frame_droppable(rtc_ref, ext_refresh_frame_flags)) return 0;
#if !CONFIG_REALTIME_ONLY
if (cpi->use_ducky_encode &&
@@ -630,10 +629,12 @@
int refresh_mask = 0;
if (ext_refresh_frame_flags->update_pending) {
- if (svc->set_ref_frame_config) {
+ if (rtc_ref->set_ref_frame_config ||
+ use_rtc_reference_structure_one_layer(cpi)) {
for (unsigned int i = 0; i < INTER_REFS_PER_FRAME; i++) {
- int ref_frame_map_idx = svc->ref_idx[i];
- refresh_mask |= svc->refresh[ref_frame_map_idx] << ref_frame_map_idx;
+ int ref_frame_map_idx = rtc_ref->ref_idx[i];
+ refresh_mask |= rtc_ref->refresh[ref_frame_map_idx]
+ << ref_frame_map_idx;
}
return refresh_mask;
}
@@ -792,6 +793,18 @@
}
if (is_second_arf) {
+ // Allocate the memory for tf_buf_second_arf buffer, only when it is
+ // required.
+ int ret = aom_realloc_frame_buffer(
+ &cpi->ppi->tf_info.tf_buf_second_arf, oxcf->frm_dim_cfg.width,
+ oxcf->frm_dim_cfg.height, cm->seq_params->subsampling_x,
+ cm->seq_params->subsampling_y, cm->seq_params->use_highbitdepth,
+ cpi->oxcf.border_in_pixels, cm->features.byte_alignment, NULL, NULL,
+ NULL, cpi->oxcf.tool_cfg.enable_global_motion, 0);
+ if (ret)
+ aom_internal_error(cm->error, AOM_CODEC_MEM_ERROR,
+ "Failed to allocate tf_buf_second_arf");
+
YV12_BUFFER_CONFIG *tf_buf_second_arf =
&cpi->ppi->tf_info.tf_buf_second_arf;
// We didn't apply temporal filtering for second arf ahead in
@@ -813,9 +826,12 @@
}
// Copy source metadata to the temporal filtered frame
- if (frame_input->source != source_buffer) {
- aom_copy_metadata_to_frame_buffer(frame_input->source,
- source_buffer->metadata);
+ if (source_buffer->metadata &&
+ aom_copy_metadata_to_frame_buffer(frame_input->source,
+ source_buffer->metadata)) {
+ aom_internal_error(
+ cm->error, AOM_CODEC_MEM_ERROR,
+ "Failed to copy source metadata to the temporal filtered frame");
}
}
#if CONFIG_COLLECT_COMPONENT_TIMING
@@ -827,8 +843,7 @@
cm->show_frame = frame_params->show_frame;
cm->current_frame.frame_type = frame_params->frame_type;
// TODO(bohanli): Why is this? what part of it is necessary?
- av1_set_frame_size(cpi, cm->superres_upscaled_width,
- cm->superres_upscaled_height);
+ av1_set_frame_size(cpi, cm->width, cm->height);
if (set_mv_params) av1_set_mv_search_params(cpi);
#if CONFIG_RD_COMMAND
@@ -980,12 +995,21 @@
#if !CONFIG_REALTIME_ONLY
if (cpi->use_ducky_encode &&
cpi->ducky_encode_info.frame_info.gop_mode == DUCKY_ENCODE_GOP_MODE_RCL) {
+ int valid_rf_idx = 0;
for (int rf = LAST_FRAME; rf < REF_FRAMES; ++rf) {
if (cpi->ppi->gf_group.ref_frame_list[gf_index][rf] != INVALID_IDX) {
remapped_ref_idx[rf - LAST_FRAME] =
cpi->ppi->gf_group.ref_frame_list[gf_index][rf];
+ valid_rf_idx = remapped_ref_idx[rf - LAST_FRAME];
}
}
+
+ for (int i = 0; i < REF_FRAMES; ++i) {
+ if (remapped_ref_idx[i] == INVALID_IDX)
+ remapped_ref_idx[i] = valid_rf_idx;
+ }
+
+ return;
}
#endif // !CONFIG_REALTIME_ONLY
@@ -1342,7 +1366,7 @@
frame_params.show_frame = 1;
} else {
source = choose_frame_source(cpi, &flush, pop_lookahead, &last_source,
- &frame_params);
+ &frame_params.show_frame);
}
if (source == NULL) { // If no source was found, we can't encode a frame.
@@ -1409,14 +1433,16 @@
start_timing(cpi, av1_get_one_pass_rt_params_time);
#endif
#if CONFIG_REALTIME_ONLY
- av1_get_one_pass_rt_params(cpi, &frame_params, &frame_input, *frame_flags);
- if (use_one_pass_rt_reference_structure(cpi))
- av1_set_reference_structure_one_pass_rt(cpi, cpi->gf_frame_index == 0);
+ av1_get_one_pass_rt_params(cpi, &frame_params.frame_type, &frame_input,
+ *frame_flags);
+ if (use_rtc_reference_structure_one_layer(cpi))
+ av1_set_rtc_reference_structure_one_layer(cpi, cpi->gf_frame_index == 0);
#else
if (use_one_pass_rt_params) {
- av1_get_one_pass_rt_params(cpi, &frame_params, &frame_input, *frame_flags);
- if (use_one_pass_rt_reference_structure(cpi))
- av1_set_reference_structure_one_pass_rt(cpi, cpi->gf_frame_index == 0);
+ av1_get_one_pass_rt_params(cpi, &frame_params.frame_type, &frame_input,
+ *frame_flags);
+ if (use_rtc_reference_structure_one_layer(cpi))
+ av1_set_rtc_reference_structure_one_layer(cpi, cpi->gf_frame_index == 0);
}
#endif
#if CONFIG_COLLECT_COMPONENT_TIMING
@@ -1479,7 +1505,6 @@
gf_group->refbuf_state[cpi->gf_frame_index], force_refresh_all);
if (!is_stat_generation_stage(cpi)) {
- const RefCntBuffer *ref_frames[INTER_REFS_PER_FRAME];
const YV12_BUFFER_CONFIG *ref_frame_buf[INTER_REFS_PER_FRAME];
RefFrameMapPair ref_frame_map_pairs[REF_FRAMES];
@@ -1498,16 +1523,24 @@
if (!ext_flags->refresh_frame.update_pending) {
av1_get_ref_frames(ref_frame_map_pairs, cur_frame_disp, cpi,
cpi->gf_frame_index, 1, cm->remapped_ref_idx);
- } else if (cpi->svc.set_ref_frame_config) {
+ } else if (cpi->rtc_ref.set_ref_frame_config ||
+ use_rtc_reference_structure_one_layer(cpi)) {
for (unsigned int i = 0; i < INTER_REFS_PER_FRAME; i++)
- cm->remapped_ref_idx[i] = cpi->svc.ref_idx[i];
+ cm->remapped_ref_idx[i] = cpi->rtc_ref.ref_idx[i];
}
}
// Get the reference frames
+ bool has_ref_frames = false;
for (int i = 0; i < INTER_REFS_PER_FRAME; ++i) {
- ref_frames[i] = get_ref_frame_buf(cm, ref_frame_priority_order[i]);
- ref_frame_buf[i] = ref_frames[i] != NULL ? &ref_frames[i]->buf : NULL;
+ const RefCntBuffer *ref_frame =
+ get_ref_frame_buf(cm, ref_frame_priority_order[i]);
+ ref_frame_buf[i] = ref_frame != NULL ? &ref_frame->buf : NULL;
+ if (ref_frame != NULL) has_ref_frames = true;
+ }
+ if (!has_ref_frames && (frame_params.frame_type == INTER_FRAME ||
+ frame_params.frame_type == S_FRAME)) {
+ return AOM_CODEC_ERROR;
}
// Work out which reference frame slots may be used.
@@ -1628,7 +1661,8 @@
// Leave a signal for a higher level caller about if this frame is droppable
if (*size > 0) {
- cpi->droppable = is_frame_droppable(&cpi->svc, &ext_flags->refresh_frame);
+ cpi->droppable =
+ is_frame_droppable(&cpi->rtc_ref, &ext_flags->refresh_frame);
}
return AOM_CODEC_OK;
diff --git a/av1/encoder/encode_strategy.h b/av1/encoder/encode_strategy.h
index 45f774d..c1d14d1 100644
--- a/av1/encoder/encode_strategy.h
+++ b/av1/encoder/encode_strategy.h
@@ -95,12 +95,12 @@
const COMPRESSOR_STAGE compressor_stage);
static AOM_INLINE int is_frame_droppable(
- const SVC *const svc,
+ const RTC_REF *const rtc_ref,
const ExtRefreshFrameFlagsInfo *const ext_refresh_frame_flags) {
// Droppable frame is only used by external refresh flags. VoD setting won't
// trigger its use case.
- if (svc->set_ref_frame_config)
- return svc->non_reference_frame;
+ if (rtc_ref->set_ref_frame_config)
+ return rtc_ref->non_reference_frame;
else if (ext_refresh_frame_flags->update_pending)
return !(ext_refresh_frame_flags->alt_ref_frame ||
ext_refresh_frame_flags->alt2_ref_frame ||
diff --git a/av1/encoder/encodeframe.c b/av1/encoder/encodeframe.c
index 2a395aa..a3862b4 100644
--- a/av1/encoder/encodeframe.c
+++ b/av1/encoder/encodeframe.c
@@ -224,7 +224,7 @@
* \param[in] mi_col Block column (in "MI_SIZE" units) index
* \param[out] num_planes Number of image planes (e.g. Y,U,V)
*
- * \return No return value but updates macroblock and thread data
+ * \remark No return value but updates macroblock and thread data
* related to the q / q delta to be used.
*/
static AOM_INLINE void setup_delta_q(AV1_COMP *const cpi, ThreadData *td,
@@ -523,15 +523,15 @@
set_cb_offsets(td->mb.cb_offset, 0, 0);
// Initialize the flag to skip cdef to 1.
+ const int block64_in_sb = (sb_size == BLOCK_128X128) ? 2 : 1;
if (sf->rt_sf.skip_cdef_sb) {
// If 128x128 block is used, we need to set the flag for all 4 64x64 sub
// "blocks".
- const int block64_in_sb = (sb_size == BLOCK_128X128) ? 2 : 1;
for (int r = 0; r < block64_in_sb; ++r) {
for (int c = 0; c < block64_in_sb; ++c) {
const int idx_in_sb =
r * MI_SIZE_64X64 * cm->mi_params.mi_stride + c * MI_SIZE_64X64;
- if (mi[idx_in_sb]) mi[idx_in_sb]->skip_cdef_curr_sb = 1;
+ if (mi[idx_in_sb]) mi[idx_in_sb]->cdef_strength = 1;
}
}
}
@@ -545,16 +545,15 @@
end_timing(cpi, nonrd_use_partition_time);
#endif
- if (sf->rt_sf.skip_cdef_sb) {
+ if (sf->rt_sf.skip_cdef_sb && block64_in_sb == 2) {
// If 128x128 block is used, we need to set the flag for all 4 64x64 sub
// "blocks".
- const int block64_in_sb = (sb_size == BLOCK_128X128) ? 2 : 1;
- const int skip = mi[0]->skip_cdef_curr_sb;
+ const int skip = mi[0]->cdef_strength;
for (int r = 0; r < block64_in_sb; ++r) {
for (int c = 0; c < block64_in_sb; ++c) {
const int idx_in_sb =
r * MI_SIZE_64X64 * cm->mi_params.mi_stride + c * MI_SIZE_64X64;
- if (mi[idx_in_sb]) mi[idx_in_sb]->skip_cdef_curr_sb = skip;
+ if (mi[idx_in_sb]) mi[idx_in_sb]->cdef_strength = skip;
}
}
}
@@ -754,31 +753,85 @@
}
}
-// Check if the cost update of symbols mode, coeff and dv are tile or off.
-static AOM_INLINE int is_mode_coeff_dv_upd_freq_tile_or_off(
- const AV1_COMP *const cpi) {
- const INTER_MODE_SPEED_FEATURES *const inter_sf = &cpi->sf.inter_sf;
+/*!\brief Calculate source SAD at superblock level using 64x64 block source SAD
+ *
+ * \ingroup partition_search
+ * \callgraph
+ * \callergraph
+ */
+static AOM_INLINE uint64_t get_sb_source_sad(const AV1_COMP *cpi, int mi_row,
+ int mi_col) {
+ if (cpi->src_sad_blk_64x64 == NULL) return UINT64_MAX;
- return (inter_sf->coeff_cost_upd_level <= INTERNAL_COST_UPD_TILE &&
- inter_sf->mode_cost_upd_level <= INTERNAL_COST_UPD_TILE &&
- cpi->sf.intra_sf.dv_cost_upd_level <= INTERNAL_COST_UPD_TILE);
+ const AV1_COMMON *const cm = &cpi->common;
+ const int blk_64x64_in_mis = (cm->seq_params->sb_size == BLOCK_128X128)
+ ? (cm->seq_params->mib_size >> 1)
+ : cm->seq_params->mib_size;
+ const int num_blk_64x64_cols =
+ (cm->mi_params.mi_cols + blk_64x64_in_mis - 1) / blk_64x64_in_mis;
+ const int num_blk_64x64_rows =
+ (cm->mi_params.mi_rows + blk_64x64_in_mis - 1) / blk_64x64_in_mis;
+ const int blk_64x64_col_index = mi_col / blk_64x64_in_mis;
+ const int blk_64x64_row_index = mi_row / blk_64x64_in_mis;
+ uint64_t curr_sb_sad = UINT64_MAX;
+ const uint64_t *const src_sad_blk_64x64_data =
+ &cpi->src_sad_blk_64x64[blk_64x64_col_index +
+ blk_64x64_row_index * num_blk_64x64_cols];
+ if (cm->seq_params->sb_size == BLOCK_128X128 &&
+ blk_64x64_col_index + 1 < num_blk_64x64_cols &&
+ blk_64x64_row_index + 1 < num_blk_64x64_rows) {
+ // Calculate SB source SAD by accumulating source SAD of 64x64 blocks in the
+ // superblock
+ curr_sb_sad = src_sad_blk_64x64_data[0] + src_sad_blk_64x64_data[1] +
+ src_sad_blk_64x64_data[num_blk_64x64_cols] +
+ src_sad_blk_64x64_data[num_blk_64x64_cols + 1];
+ } else if (cm->seq_params->sb_size == BLOCK_64X64) {
+ curr_sb_sad = src_sad_blk_64x64_data[0];
+ }
+ return curr_sb_sad;
}
-// When row-mt is enabled and cost update frequencies are set to off/tile,
-// processing of current SB can start even before processing of top-right SB
-// is finished. This function checks if it is sufficient to wait for top SB
-// to finish processing before current SB starts processing.
-static AOM_INLINE int delay_wait_for_top_right_sb(const AV1_COMP *const cpi) {
- const MODE mode = cpi->oxcf.mode;
- if (mode == GOOD) return 0;
+/*!\brief Determine whether grading content can be skipped based on sad stat
+ *
+ * \ingroup partition_search
+ * \callgraph
+ * \callergraph
+ */
+static AOM_INLINE bool is_calc_src_content_needed(AV1_COMP *cpi,
+ MACROBLOCK *const x,
+ int mi_row, int mi_col) {
+ const uint64_t curr_sb_sad = get_sb_source_sad(cpi, mi_row, mi_col);
+ if (curr_sb_sad == UINT64_MAX) return true;
+ if (curr_sb_sad == 0) {
+ x->content_state_sb.source_sad_nonrd = kZeroSad;
+ return false;
+ }
+ AV1_COMMON *const cm = &cpi->common;
+ bool do_calc_src_content = true;
- if (mode == ALLINTRA)
- return is_mode_coeff_dv_upd_freq_tile_or_off(cpi);
- else if (mode == REALTIME)
- return (is_mode_coeff_dv_upd_freq_tile_or_off(cpi) &&
- cpi->sf.inter_sf.mv_cost_upd_level <= INTERNAL_COST_UPD_TILE);
- else
- return 0;
+ if (cpi->oxcf.speed < 9) return do_calc_src_content;
+
+ // TODO(yunqing): Tune/validate the thresholds for 128x128 SB size.
+ if (AOMMIN(cm->width, cm->height) < 360) {
+ // Derive Average 64x64 block source SAD from SB source SAD
+ const uint64_t avg_64x64_blk_sad =
+ (cm->seq_params->sb_size == BLOCK_128X128) ? ((curr_sb_sad + 2) >> 2)
+ : curr_sb_sad;
+
+ // The threshold is determined based on kLowSad and kHighSad threshold and
+ // test results.
+ const uint64_t thresh_low = 15000;
+ const uint64_t thresh_high = 40000;
+
+ if (avg_64x64_blk_sad > thresh_low && avg_64x64_blk_sad < thresh_high) {
+ do_calc_src_content = false;
+ // Note: set x->content_state_sb.source_sad_rd as well if this is extended
+ // to RTC rd path.
+ x->content_state_sb.source_sad_nonrd = kMedSad;
+ }
+ }
+
+ return do_calc_src_content;
}
/*!\brief Determine whether grading content is needed based on sf and frame stat
@@ -789,18 +842,20 @@
*/
// TODO(any): consolidate sfs to make interface cleaner
static AOM_INLINE void grade_source_content_sb(AV1_COMP *cpi,
- MACROBLOCK *const x, int mi_row,
- int mi_col) {
+ MACROBLOCK *const x,
+ TileDataEnc *tile_data,
+ int mi_row, int mi_col) {
AV1_COMMON *const cm = &cpi->common;
bool calc_src_content = false;
if (cpi->sf.rt_sf.source_metrics_sb_nonrd &&
cpi->svc.number_spatial_layers <= 1 &&
cm->current_frame.frame_type != KEY_FRAME) {
- if (!cpi->sf.rt_sf.check_scene_detection || cpi->rc.frame_source_sad > 0)
- calc_src_content = true;
- else
+ if (!cpi->sf.rt_sf.check_scene_detection || cpi->rc.frame_source_sad > 0) {
+ calc_src_content = is_calc_src_content_needed(cpi, x, mi_row, mi_col);
+ } else {
x->content_state_sb.source_sad_nonrd = kZeroSad;
+ }
} else if ((cpi->sf.rt_sf.var_part_based_on_qidx >= 1) &&
(cm->width * cm->height <= 352 * 288)) {
if (cpi->rc.frame_source_sad > 0)
@@ -808,7 +863,8 @@
else
x->content_state_sb.source_sad_rd = kZeroSad;
}
- if (calc_src_content) av1_source_content_sb(cpi, x, mi_row, mi_col);
+ if (calc_src_content)
+ av1_source_content_sb(cpi, x, tile_data, mi_row, mi_col);
}
/*!\brief Encode a superblock row by breaking it into superblocks
@@ -895,7 +951,7 @@
x->content_state_sb.source_sad_rd = kMedSad;
x->content_state_sb.lighting_change = 0;
x->content_state_sb.low_sumdiff = 0;
- x->force_zeromv_skip = 0;
+ x->force_zeromv_skip_for_sb = 0;
if (cpi->oxcf.mode == ALLINTRA) {
x->intra_sb_rdmult_modifier = 128;
@@ -924,7 +980,7 @@
// Grade the temporal variation of the sb, the grade will be used to decide
// fast mode search strategy for coding blocks
- grade_source_content_sb(cpi, x, mi_row, mi_col);
+ grade_source_content_sb(cpi, x, tile_data, mi_row, mi_col);
// encode the superblock
if (use_nonrd_mode) {
@@ -1370,6 +1426,43 @@
#endif // !CONFIG_REALTIME_ONLY
}
+#define FORCE_ZMV_SKIP_128X128_BLK_DIFF 10000
+#define FORCE_ZMV_SKIP_MAX_PER_PIXEL_DIFF 4
+
+// Populates block level thresholds for force zeromv-skip decision
+static void populate_thresh_to_force_zeromv_skip(AV1_COMP *cpi) {
+ if (cpi->sf.rt_sf.part_early_exit_zeromv == 0) return;
+
+ // Threshold for forcing zeromv-skip decision is as below:
+ // For 128x128 blocks, threshold is 10000 and per pixel threshold is 0.6103.
+ // For 64x64 blocks, threshold is 5000 and per pixel threshold is 1.221
+ // allowing slightly higher error for smaller blocks.
+ // Per Pixel Threshold of 64x64 block Area of 64x64 block 1 1
+ // ------------------------------------=sqrt(---------------------)=sqrt(-)=-
+ // Per Pixel Threshold of 128x128 block Area of 128x128 block 4 2
+ // Thus, per pixel thresholds for blocks of size 32x32, 16x16,... can be
+ // chosen as 2.442, 4.884,.... As the per pixel error tends to be higher for
+ // small blocks, the same is clipped to 4.
+ const unsigned int thresh_exit_128x128_part = FORCE_ZMV_SKIP_128X128_BLK_DIFF;
+ const int num_128x128_pix =
+ block_size_wide[BLOCK_128X128] * block_size_high[BLOCK_128X128];
+
+ for (BLOCK_SIZE bsize = BLOCK_4X4; bsize < BLOCK_SIZES_ALL; bsize++) {
+ const int num_block_pix = block_size_wide[bsize] * block_size_high[bsize];
+
+ // Calculate the threshold for zeromv-skip decision based on area of the
+ // partition
+ unsigned int thresh_exit_part_blk =
+ (unsigned int)(thresh_exit_128x128_part *
+ sqrt((double)num_block_pix / num_128x128_pix) +
+ 0.5);
+ thresh_exit_part_blk = AOMMIN(
+ thresh_exit_part_blk,
+ (unsigned int)(FORCE_ZMV_SKIP_MAX_PER_PIXEL_DIFF * num_block_pix));
+ cpi->zeromv_skip_thresh_exit_part[bsize] = thresh_exit_part_blk;
+ }
+}
+
/*!\brief Encoder setup(only for the current frame), encoding, and recontruction
* for a single frame
*
@@ -1633,6 +1726,7 @@
// has to be called after 'skip_mode_flag' is initialized.
av1_initialize_rd_consts(cpi);
av1_set_sad_per_bit(cpi, &x->sadperbit, quant_params->base_qindex);
+ populate_thresh_to_force_zeromv_skip(cpi);
enc_row_mt->sync_read_ptr = av1_row_mt_sync_read_dummy;
enc_row_mt->sync_write_ptr = av1_row_mt_sync_write_dummy;
@@ -1927,9 +2021,10 @@
FeatureFlags *const features = &cm->features;
const int num_planes = av1_num_planes(cm);
RD_COUNTS *const rdc = &cpi->td.rd_counts;
+ const AV1EncoderConfig *const oxcf = &cpi->oxcf;
// Indicates whether or not to use a default reduced set for ext-tx
// rather than the potential full set of 16 transforms
- features->reduced_tx_set_used = cpi->oxcf.txfm_cfg.reduced_tx_type_set;
+ features->reduced_tx_set_used = oxcf->txfm_cfg.reduced_tx_type_set;
// Make sure segment_id is no larger than last_active_segid.
if (cm->seg.enabled && cm->seg.update_map) {
@@ -1971,7 +2066,8 @@
features->interp_filter = SWITCHABLE;
if (cm->tiles.large_scale) features->interp_filter = EIGHTTAP_REGULAR;
- features->switchable_motion_mode = 1;
+ features->switchable_motion_mode = is_switchable_motion_mode_allowed(
+ features->allow_warped_motion, oxcf->motion_mode_cfg.enable_obmc);
rdc->compound_ref_used_flag = 0;
rdc->skip_mode_used_flag = 0;
diff --git a/av1/encoder/encodeframe_utils.c b/av1/encoder/encodeframe_utils.c
index e011b11..75e0fcc 100644
--- a/av1/encoder/encodeframe_utils.c
+++ b/av1/encoder/encodeframe_utils.c
@@ -40,6 +40,7 @@
for (col = mi_col / num_mi_h;
col < num_cols && col < mi_col / num_mi_h + num_bcols; ++col) {
const int index = row * num_cols + col;
+ assert(cpi->ssim_rdmult_scaling_factors[index] != 0.0);
geom_mean_of_scale += log(cpi->ssim_rdmult_scaling_factors[index]);
num_of_mi += 1.0;
}
@@ -950,8 +951,10 @@
TplDepStats *this_stats = &tpl_stats[av1_tpl_ptr_pos(
row, col, tpl_stride, tpl_data->tpl_stats_block_mis_log2)];
- sb_enc->tpl_inter_cost[count] = this_stats->inter_cost;
- sb_enc->tpl_intra_cost[count] = this_stats->intra_cost;
+ sb_enc->tpl_inter_cost[count] = this_stats->inter_cost
+ << TPL_DEP_COST_SCALE_LOG2;
+ sb_enc->tpl_intra_cost[count] = this_stats->intra_cost
+ << TPL_DEP_COST_SCALE_LOG2;
memcpy(sb_enc->tpl_mv[count], this_stats->mv, sizeof(this_stats->mv));
mi_count++;
count++;
@@ -1020,7 +1023,7 @@
mc_dep_reg += log(3 * dist_scaled + mc_dep_delta) * cbcmp;
srcrf_dist += (double)(this_stats->srcrf_dist << RDDIV_BITS);
srcrf_sse += (double)(this_stats->srcrf_sse << RDDIV_BITS);
- srcrf_rate += (double)this_stats->srcrf_rate;
+ srcrf_rate += (double)(this_stats->srcrf_rate << TPL_DEP_COST_SCALE_LOG2);
#ifndef NDEBUG
mi_count++;
#endif
@@ -1309,10 +1312,78 @@
CFL_ALPHABET_SIZE);
}
+// Check neighbor blocks' motion information.
+static int check_neighbor_blocks(MB_MODE_INFO **mi, int mi_stride,
+ const TileInfo *const tile_info, int mi_row,
+ int mi_col) {
+ int is_above_low_motion = 1;
+ int is_left_low_motion = 1;
+ const int thr = 24;
+
+ // Check above block.
+ if (mi_row > tile_info->mi_row_start) {
+ const MB_MODE_INFO *above_mbmi = mi[-mi_stride];
+ const int_mv above_mv = above_mbmi->mv[0];
+ if (above_mbmi->mode >= INTRA_MODE_END &&
+ (abs(above_mv.as_mv.row) > thr || abs(above_mv.as_mv.col) > thr))
+ is_above_low_motion = 0;
+ }
+
+ // Check left block.
+ if (mi_col > tile_info->mi_col_start) {
+ const MB_MODE_INFO *left_mbmi = mi[-1];
+ const int_mv left_mv = left_mbmi->mv[0];
+ if (left_mbmi->mode >= INTRA_MODE_END &&
+ (abs(left_mv.as_mv.row) > thr || abs(left_mv.as_mv.col) > thr))
+ is_left_low_motion = 0;
+ }
+
+ return (is_above_low_motion && is_left_low_motion);
+}
+
+// Check this block's motion in a fast way.
+static int fast_detect_non_zero_motion(AV1_COMP *cpi, const uint8_t *src_y,
+ int src_ystride,
+ const uint8_t *last_src_y,
+ int last_src_ystride, int mi_row,
+ int mi_col) {
+ AV1_COMMON *const cm = &cpi->common;
+ const BLOCK_SIZE bsize = cm->seq_params->sb_size;
+ unsigned int blk_sad = INT_MAX;
+ if (cpi->src_sad_blk_64x64 != NULL) {
+ const int sb_size_by_mb = (bsize == BLOCK_128X128)
+ ? (cm->seq_params->mib_size >> 1)
+ : cm->seq_params->mib_size;
+ const int sb_cols =
+ (cm->mi_params.mi_cols + sb_size_by_mb - 1) / sb_size_by_mb;
+ const int sbi_col = mi_col / sb_size_by_mb;
+ const int sbi_row = mi_row / sb_size_by_mb;
+ blk_sad = (unsigned int)cpi->src_sad_blk_64x64[sbi_col + sbi_row * sb_cols];
+ } else {
+ blk_sad = cpi->ppi->fn_ptr[bsize].sdf(src_y, src_ystride, last_src_y,
+ last_src_ystride);
+ }
+
+ // Search 4 1-away points.
+ const uint8_t *const search_pos[4] = {
+ last_src_y - last_src_ystride,
+ last_src_y - 1,
+ last_src_y + 1,
+ last_src_y + last_src_ystride,
+ };
+ unsigned int sad_arr[4];
+ cpi->ppi->fn_ptr[bsize].sdx4df(src_y, src_ystride, search_pos,
+ last_src_ystride, sad_arr);
+
+ blk_sad = (blk_sad * 5) >> 3;
+ return (blk_sad < sad_arr[0] && blk_sad < sad_arr[1] &&
+ blk_sad < sad_arr[2] && blk_sad < sad_arr[3]);
+}
+
// Grade the temporal variation of the source by comparing the current sb and
// its collocated block in the last frame.
-void av1_source_content_sb(AV1_COMP *cpi, MACROBLOCK *x, int mi_row,
- int mi_col) {
+void av1_source_content_sb(AV1_COMP *cpi, MACROBLOCK *x, TileDataEnc *tile_data,
+ int mi_row, int mi_col) {
unsigned int tmp_sse;
unsigned int tmp_variance;
const BLOCK_SIZE bsize = cpi->common.seq_params->sb_size;
@@ -1321,8 +1392,10 @@
uint8_t *last_src_y = cpi->last_source->y_buffer;
int last_src_ystride = cpi->last_source->y_stride;
const int offset = cpi->source->y_stride * (mi_row << 2) + (mi_col << 2);
- uint64_t avg_source_sse_threshold[2] = { 100000, // ~5*5*(64*64)
- 36000 }; // ~3*3*(64*64)
+ uint64_t avg_source_sse_threshold_verylow = 10000; // ~1.5*1.5*(64*64)
+ uint64_t avg_source_sse_threshold_low[2] = { 100000, // ~5*5*(64*64)
+ 36000 }; // ~3*3*(64*64)
+
uint64_t avg_source_sse_threshold_high = 1000000; // ~15*15*(64*64)
uint64_t sum_sq_thresh = 10000; // sum = sqrt(thresh / 64*64)) ~1.5
#if CONFIG_AV1_HIGHBITDEPTH
@@ -1334,13 +1407,15 @@
tmp_variance = cpi->ppi->fn_ptr[bsize].vf(src_y, src_ystride, last_src_y,
last_src_ystride, &tmp_sse);
// rd thresholds
- if (tmp_sse < avg_source_sse_threshold[1])
+ if (tmp_sse < avg_source_sse_threshold_low[1])
x->content_state_sb.source_sad_rd = kLowSad;
// nonrd thresholds
if (tmp_sse == 0)
x->content_state_sb.source_sad_nonrd = kZeroSad;
- else if (tmp_sse < avg_source_sse_threshold[0])
+ else if (tmp_sse < avg_source_sse_threshold_verylow)
+ x->content_state_sb.source_sad_nonrd = kVeryLowSad;
+ else if (tmp_sse < avg_source_sse_threshold_low[0])
x->content_state_sb.source_sad_nonrd = kLowSad;
else if (tmp_sse > avg_source_sse_threshold_high)
x->content_state_sb.source_sad_nonrd = kHighSad;
@@ -1358,7 +1433,9 @@
if (cpi->last_source->y_width != cpi->source->y_width ||
cpi->last_source->y_height != cpi->source->y_height)
return;
- if (!cpi->sf.rt_sf.use_rtc_tf) return;
+ if (!cpi->sf.rt_sf.use_rtc_tf || tmp_sse == 0 || cpi->rc.high_source_sad ||
+ cpi->rc.frame_source_sad > 20000)
+ return;
// In-place temporal filter. If psnr calculation is enabled, we store the
// source for that.
@@ -1367,10 +1444,35 @@
const unsigned int nmean2 = tmp_sse - tmp_variance;
const int ac_q_step = av1_ac_quant_QTX(cm->quant_params.base_qindex, 0,
cm->seq_params->bit_depth);
- const unsigned int threshold = 3 * ac_q_step * ac_q_step / 2;
+ const PRIMARY_RATE_CONTROL *const p_rc = &cpi->ppi->p_rc;
+ const int avg_q_step = av1_ac_quant_QTX(p_rc->avg_frame_qindex[INTER_FRAME],
+ 0, cm->seq_params->bit_depth);
+
+ const unsigned int threshold =
+ (cpi->sf.rt_sf.use_rtc_tf == 1)
+ ? (clamp(avg_q_step, 250, 1000)) * ac_q_step
+ : 250 * ac_q_step;
// TODO(yunqing): use a weighted sum instead of averaging in filtering.
if (tmp_variance <= threshold && nmean2 <= 15) {
+ // Check neighbor blocks. If neighbor blocks aren't low-motion blocks,
+ // skip temporal filtering for this block.
+ MB_MODE_INFO **mi = cm->mi_params.mi_grid_base +
+ get_mi_grid_idx(&cm->mi_params, mi_row, mi_col);
+ const TileInfo *const tile_info = &tile_data->tile_info;
+ const int is_neighbor_blocks_low_motion = check_neighbor_blocks(
+ mi, cm->mi_params.mi_stride, tile_info, mi_row, mi_col);
+ if (!is_neighbor_blocks_low_motion) return;
+
+ // Only consider 64x64 SB for now. Need to extend to 128x128 for large SB
+ // size.
+ // Test several nearby points. If non-zero mv exists, don't do temporal
+ // filtering.
+ const int is_this_blk_low_motion = fast_detect_non_zero_motion(
+ cpi, src_y, src_ystride, last_src_y, last_src_ystride, mi_row, mi_col);
+
+ if (!is_this_blk_low_motion) return;
+
const int shift_x[2] = { 0, cpi->source->subsampling_x };
const int shift_y[2] = { 0, cpi->source->subsampling_y };
const uint8_t h = block_size_high[bsize];
diff --git a/av1/encoder/encodeframe_utils.h b/av1/encoder/encodeframe_utils.h
index 3a0df60..0e9c439 100644
--- a/av1/encoder/encodeframe_utils.h
+++ b/av1/encoder/encodeframe_utils.h
@@ -391,8 +391,8 @@
void av1_avg_cdf_symbols(FRAME_CONTEXT *ctx_left, FRAME_CONTEXT *ctx_tr,
int wt_left, int wt_tr);
-void av1_source_content_sb(AV1_COMP *cpi, MACROBLOCK *x, int mi_row,
- int mi_col);
+void av1_source_content_sb(AV1_COMP *cpi, MACROBLOCK *x, TileDataEnc *tile_data,
+ int mi_row, int mi_col);
void av1_reset_mbmi(CommonModeInfoParams *const mi_params, BLOCK_SIZE sb_size,
int mi_row, int mi_col);
@@ -565,6 +565,39 @@
assert(total_valid_refs <= max_allowed_refs);
}
+// Check if the cost update of symbols mode, coeff and dv are tile or off.
+static AOM_INLINE int is_mode_coeff_dv_upd_freq_tile_or_off(
+ const AV1_COMP *const cpi) {
+ const INTER_MODE_SPEED_FEATURES *const inter_sf = &cpi->sf.inter_sf;
+
+ return (inter_sf->coeff_cost_upd_level <= INTERNAL_COST_UPD_TILE &&
+ inter_sf->mode_cost_upd_level <= INTERNAL_COST_UPD_TILE &&
+ cpi->sf.intra_sf.dv_cost_upd_level <= INTERNAL_COST_UPD_TILE);
+}
+
+// When row-mt is enabled and cost update frequencies are set to off/tile,
+// processing of current SB can start even before processing of top-right SB
+// is finished. This function checks if it is sufficient to wait for top SB
+// to finish processing before current SB starts processing.
+static AOM_INLINE int delay_wait_for_top_right_sb(const AV1_COMP *const cpi) {
+ const MODE mode = cpi->oxcf.mode;
+ if (mode == GOOD) return 0;
+
+ if (mode == ALLINTRA)
+ return is_mode_coeff_dv_upd_freq_tile_or_off(cpi);
+ else if (mode == REALTIME)
+ return (is_mode_coeff_dv_upd_freq_tile_or_off(cpi) &&
+ cpi->sf.inter_sf.mv_cost_upd_level <= INTERNAL_COST_UPD_TILE);
+ else
+ return 0;
+}
+
+// This function checks if top right dependency wait at mi level can be enabled.
+static AOM_INLINE int enable_top_right_sync_wait_in_mis(const AV1_COMP *cpi,
+ int seg_skip_active) {
+ return cpi->sf.rt_sf.top_right_sync_wait_in_mis && !seg_skip_active &&
+ delay_wait_for_top_right_sb(cpi);
+}
#ifdef __cplusplus
} // extern "C"
#endif
diff --git a/av1/encoder/encodemb.h b/av1/encoder/encodemb.h
index b58d13d..b819e82 100644
--- a/av1/encoder/encodemb.h
+++ b/av1/encoder/encodemb.h
@@ -56,7 +56,7 @@
const struct AV1_COMP *cpi;
MACROBLOCK *x;
struct optimize_ctx *ctx;
- int8_t *skip;
+ uint8_t *skip;
ENTROPY_CONTEXT *ta;
ENTROPY_CONTEXT *tl;
RUN_TYPE dry_run;
diff --git a/av1/encoder/encoder.c b/av1/encoder/encoder.c
index 3dfe0cf..7163d91 100644
--- a/av1/encoder/encoder.c
+++ b/av1/encoder/encoder.c
@@ -94,33 +94,33 @@
FILE *yuv_denoised_file = NULL;
#endif
-static INLINE void Scale2Ratio(AOM_SCALING mode, int *hr, int *hs) {
+static INLINE void Scale2Ratio(AOM_SCALING_MODE mode, int *hr, int *hs) {
switch (mode) {
- case NORMAL:
+ case AOME_NORMAL:
*hr = 1;
*hs = 1;
break;
- case FOURFIVE:
+ case AOME_FOURFIVE:
*hr = 4;
*hs = 5;
break;
- case THREEFIVE:
+ case AOME_THREEFIVE:
*hr = 3;
*hs = 5;
break;
- case THREEFOUR:
+ case AOME_THREEFOUR:
*hr = 3;
*hs = 4;
break;
- case ONEFOUR:
+ case AOME_ONEFOUR:
*hr = 1;
*hs = 4;
break;
- case ONEEIGHT:
+ case AOME_ONEEIGHT:
*hr = 1;
*hs = 8;
break;
- case ONETWO:
+ case AOME_ONETWO:
*hr = 1;
*hs = 2;
break;
@@ -136,30 +136,27 @@
int cols) {
const CommonModeInfoParams *const mi_params = &cpi->common.mi_params;
if (rows == mi_params->mb_rows && cols == mi_params->mb_cols) {
- unsigned char *const active_map_8x8 = cpi->active_map.map;
+ unsigned char *const active_map_4x4 = cpi->active_map.map;
const int mi_rows = mi_params->mi_rows;
const int mi_cols = mi_params->mi_cols;
const int row_scale = mi_size_high[BLOCK_16X16] == 2 ? 1 : 2;
const int col_scale = mi_size_wide[BLOCK_16X16] == 2 ? 1 : 2;
- cpi->active_map.update = 1;
+ cpi->active_map.update = 0;
if (new_map_16x16) {
- int r, c;
- for (r = 0; r < mi_rows; ++r) {
- for (c = 0; c < mi_cols; ++c) {
- active_map_8x8[r * mi_cols + c] =
+ for (int r = 0; r < mi_rows; ++r) {
+ for (int c = 0; c < mi_cols; ++c) {
+ active_map_4x4[r * mi_cols + c] =
new_map_16x16[(r >> row_scale) * cols + (c >> col_scale)]
? AM_SEGMENT_ID_ACTIVE
: AM_SEGMENT_ID_INACTIVE;
}
}
cpi->active_map.enabled = 1;
- } else {
- cpi->active_map.enabled = 0;
}
return 0;
- } else {
- return -1;
}
+
+ return -1;
}
int av1_get_active_map(AV1_COMP *cpi, unsigned char *new_map_16x16, int rows,
@@ -175,9 +172,8 @@
memset(new_map_16x16, !cpi->active_map.enabled, rows * cols);
if (cpi->active_map.enabled) {
- int r, c;
- for (r = 0; r < mi_rows; ++r) {
- for (c = 0; c < mi_cols; ++c) {
+ for (int r = 0; r < mi_rows; ++r) {
+ for (int c = 0; c < mi_cols; ++c) {
// Cyclic refresh segments are considered active despite not having
// AM_SEGMENT_ID_ACTIVE
new_map_16x16[(r >> row_scale) * cols + (c >> col_scale)] |=
@@ -186,9 +182,9 @@
}
}
return 0;
- } else {
- return -1;
}
+
+ return -1;
}
void av1_initialize_enc(unsigned int usage, enum aom_rc_mode end_usage) {
@@ -224,6 +220,36 @@
return uncompressed_frame_size / (double)encoded_frame_size;
}
+static void auto_tile_size_balancing(AV1_COMMON *const cm, int num_sbs,
+ int num_tiles_lg, int tile_col_row) {
+ CommonTileParams *const tiles = &cm->tiles;
+ int i, start_sb;
+ int size_sb = num_sbs >> num_tiles_lg;
+ int res_sbs = num_sbs - (size_sb << num_tiles_lg);
+ int num_tiles = 1 << num_tiles_lg;
+ int inc_index = num_tiles - res_sbs;
+
+ tiles->uniform_spacing = 0;
+
+ for (i = 0, start_sb = 0; start_sb < num_sbs && i < MAX_TILE_COLS; ++i) {
+ if (i == inc_index) ++size_sb;
+ if (tile_col_row)
+ tiles->col_start_sb[i] = start_sb;
+ else
+ tiles->row_start_sb[i] = start_sb;
+
+ start_sb += AOMMIN(size_sb, tiles->max_width_sb);
+ }
+
+ if (tile_col_row) {
+ tiles->cols = i;
+ tiles->col_start_sb[i] = num_sbs;
+ } else {
+ tiles->rows = i;
+ tiles->row_start_sb[i] = num_sbs;
+ }
+}
+
static void set_tile_info(AV1_COMMON *const cm,
const TileConfig *const tile_cfg) {
const CommonModeInfoParams *const mi_params = &cm->mi_params;
@@ -233,14 +259,16 @@
av1_get_tile_limits(cm);
+ int sb_cols =
+ CEIL_POWER_OF_TWO(mi_params->mi_cols, seq_params->mib_size_log2);
// configure tile columns
if (tile_cfg->tile_width_count == 0 || tile_cfg->tile_height_count == 0) {
tiles->uniform_spacing = 1;
tiles->log2_cols = AOMMAX(tile_cfg->tile_columns, tiles->min_log2_cols);
tiles->log2_cols = AOMMIN(tiles->log2_cols, tiles->max_log2_cols);
+ } else if (tile_cfg->tile_widths[0] < 0) {
+ auto_tile_size_balancing(cm, sb_cols, tile_cfg->tile_columns, 1);
} else {
- int sb_cols =
- CEIL_POWER_OF_TWO(mi_params->mi_cols, seq_params->mib_size_log2);
int size_sb, j = 0;
tiles->uniform_spacing = 0;
for (i = 0, start_sb = 0; start_sb < sb_cols && i < MAX_TILE_COLS; i++) {
@@ -256,12 +284,14 @@
tiles);
// configure tile rows
+ int sb_rows =
+ CEIL_POWER_OF_TWO(mi_params->mi_rows, seq_params->mib_size_log2);
if (tiles->uniform_spacing) {
tiles->log2_rows = AOMMAX(tile_cfg->tile_rows, tiles->min_log2_rows);
tiles->log2_rows = AOMMIN(tiles->log2_rows, tiles->max_log2_rows);
+ } else if (tile_cfg->tile_heights[0] < 0) {
+ auto_tile_size_balancing(cm, sb_rows, tile_cfg->tile_rows, 0);
} else {
- int sb_rows =
- CEIL_POWER_OF_TWO(mi_params->mi_rows, seq_params->mib_size_log2);
int size_sb, j = 0;
for (i = 0, start_sb = 0; start_sb < sb_rows && i < MAX_TILE_ROWS; i++) {
tiles->row_start_sb[i] = start_sb;
@@ -543,18 +573,20 @@
alloc_compressor_data(cpi);
- av1_update_film_grain_parameters(cpi, oxcf);
-
// Single thread case: use counts in common.
cpi->td.counts = &cpi->counts;
- // Set init SVC parameters.
- cpi->svc.set_ref_frame_config = 0;
- cpi->svc.non_reference_frame = 0;
+ // Init SVC parameters.
cpi->svc.number_spatial_layers = 1;
cpi->svc.number_temporal_layers = 1;
cm->spatial_layer_id = 0;
cm->temporal_layer_id = 0;
+ // Init rtc_ref parameters.
+ cpi->rtc_ref.set_ref_frame_config = 0;
+ cpi->rtc_ref.non_reference_frame = 0;
+ cpi->rtc_ref.ref_frame_comp[0] = 0;
+ cpi->rtc_ref.ref_frame_comp[1] = 0;
+ cpi->rtc_ref.ref_frame_comp[2] = 0;
// change includes all joint functionality
av1_change_config(cpi, oxcf, false);
@@ -664,6 +696,7 @@
RefreshFrameInfo *const refresh_frame = &cpi->refresh_frame;
const FrameDimensionCfg *const frm_dim_cfg = &cpi->oxcf.frm_dim_cfg;
const RateControlCfg *const rc_cfg = &oxcf->rc_cfg;
+ FeatureFlags *const features = &cm->features;
// in case of LAP, lag in frames is set according to number of lap buffers
// calculated at init time. This stores and restores LAP's lag in frames to
@@ -673,9 +706,10 @@
lap_lag_in_frames = cpi->oxcf.gf_cfg.lag_in_frames;
}
+ cpi->oxcf = *oxcf;
+
av1_update_film_grain_parameters(cpi, oxcf);
- cpi->oxcf = *oxcf;
// When user provides superres_mode = AOM_SUPERRES_AUTO, we still initialize
// superres mode for current encoding = AOM_SUPERRES_NONE. This is to ensure
// that any analysis (e.g. TPL) happening outside the main encoding loop still
@@ -717,12 +751,12 @@
refresh_frame->golden_frame = false;
refresh_frame->bwd_ref_frame = false;
- cm->features.refresh_frame_context =
+ features->refresh_frame_context =
(oxcf->tool_cfg.frame_parallel_decoding_mode)
? REFRESH_FRAME_CONTEXT_DISABLED
: REFRESH_FRAME_CONTEXT_BACKWARD;
if (oxcf->tile_cfg.enable_large_scale_tile)
- cm->features.refresh_frame_context = REFRESH_FRAME_CONTEXT_DISABLED;
+ features->refresh_frame_context = REFRESH_FRAME_CONTEXT_DISABLED;
if (x->palette_buffer == NULL) {
CHECK_MEM_ERROR(cm, x->palette_buffer,
@@ -770,9 +804,10 @@
rc->worst_quality = rc_cfg->worst_allowed_q;
rc->best_quality = rc_cfg->best_allowed_q;
- cm->features.interp_filter =
+ features->interp_filter =
oxcf->tile_cfg.enable_large_scale_tile ? EIGHTTAP_REGULAR : SWITCHABLE;
- cm->features.switchable_motion_mode = 1;
+ features->switchable_motion_mode = is_switchable_motion_mode_allowed(
+ features->allow_warped_motion, oxcf->motion_mode_cfg.enable_obmc);
if (frm_dim_cfg->render_width > 0 && frm_dim_cfg->render_height > 0) {
cm->render_width = frm_dim_cfg->render_width;
@@ -803,7 +838,7 @@
set_tile_info(cm, &cpi->oxcf.tile_cfg);
- if (!cpi->svc.set_ref_frame_config)
+ if (!cpi->rtc_ref.set_ref_frame_config)
cpi->ext_flags.refresh_frame.update_pending = 0;
cpi->ext_flags.refresh_frame_context_pending = 0;
@@ -1210,13 +1245,14 @@
BufferPool *const pool, COMPRESSOR_STAGE stage,
int lap_lag_in_frames) {
AV1_COMP *volatile const cpi = aom_memalign(32, sizeof(AV1_COMP));
- AV1_COMMON *volatile const cm = cpi != NULL ? &cpi->common : NULL;
- if (!cm) return NULL;
+ if (!cpi) return NULL;
av1_zero(*cpi);
cpi->ppi = ppi;
+
+ AV1_COMMON *volatile const cm = &cpi->common;
cm->seq_params = &ppi->seq_params;
cm->error =
(struct aom_internal_error_info *)aom_calloc(1, sizeof(*cm->error));
@@ -1335,8 +1371,17 @@
av1_set_speed_features_framesize_independent(cpi, oxcf->speed);
av1_set_speed_features_framesize_dependent(cpi, oxcf->speed);
+ int max_mi_cols = mi_params->mi_cols;
+ int max_mi_rows = mi_params->mi_rows;
+ if (oxcf->frm_dim_cfg.forced_max_frame_width) {
+ max_mi_cols = size_in_mi(oxcf->frm_dim_cfg.forced_max_frame_width);
+ }
+ if (oxcf->frm_dim_cfg.forced_max_frame_height) {
+ max_mi_rows = size_in_mi(oxcf->frm_dim_cfg.forced_max_frame_height);
+ }
+
CHECK_MEM_ERROR(cm, cpi->consec_zero_mv,
- aom_calloc((mi_params->mi_rows * mi_params->mi_cols) >> 2,
+ aom_calloc((max_mi_rows * max_mi_cols) >> 2,
sizeof(*cpi->consec_zero_mv)));
cpi->mb_weber_stats = NULL;
@@ -1346,8 +1391,8 @@
const int bsize = BLOCK_16X16;
const int w = mi_size_wide[bsize];
const int h = mi_size_high[bsize];
- const int num_cols = (mi_params->mi_cols + w - 1) / w;
- const int num_rows = (mi_params->mi_rows + h - 1) / h;
+ const int num_cols = (max_mi_cols + w - 1) / w;
+ const int num_rows = (max_mi_rows + h - 1) / h;
CHECK_MEM_ERROR(cm, cpi->ssim_rdmult_scaling_factors,
aom_calloc(num_rows * num_cols,
sizeof(*cpi->ssim_rdmult_scaling_factors)));
@@ -1918,14 +1963,22 @@
static void init_ref_frame_bufs(AV1_COMP *cpi) {
AV1_COMMON *const cm = &cpi->common;
int i;
- BufferPool *const pool = cm->buffer_pool;
- cm->cur_frame = NULL;
+ if (cm->cur_frame) {
+ cm->cur_frame->ref_count--;
+ cm->cur_frame = NULL;
+ }
for (i = 0; i < REF_FRAMES; ++i) {
- cm->ref_frame_map[i] = NULL;
+ if (cm->ref_frame_map[i]) {
+ cm->ref_frame_map[i]->ref_count--;
+ cm->ref_frame_map[i] = NULL;
+ }
}
+#ifndef NDEBUG
+ BufferPool *const pool = cm->buffer_pool;
for (i = 0; i < FRAME_BUFFERS; ++i) {
- pool->frame_bufs[i].ref_count = 0;
+ assert(pool->frame_bufs[i].ref_count == 0);
}
+#endif
}
void av1_check_initial_width(AV1_COMP *cpi, int use_highbitdepth,
@@ -2108,6 +2161,28 @@
set_ref_ptrs(cm, xd, LAST_FRAME, LAST_FRAME);
}
+static INLINE int extend_borders_mt(const AV1_COMP *cpi,
+ MULTI_THREADED_MODULES stage, int plane) {
+ const AV1_COMMON *const cm = &cpi->common;
+ if (cpi->mt_info.num_mod_workers[stage] < 2) return 0;
+ switch (stage) {
+ // TODO(deepa.kg@ittiam.com): When cdef and loop-restoration are disabled,
+ // multi-thread frame border extension along with loop filter frame.
+ // As loop-filtering of a superblock row modifies the pixels of the
+ // above superblock row, border extension requires that loop filtering
+ // of the current and above superblock row is complete.
+ case MOD_LPF: return 0;
+ case MOD_CDEF:
+ return is_cdef_used(cm) && !cpi->rtc_ref.non_reference_frame &&
+ !is_restoration_used(cm) && !av1_superres_scaled(cm);
+ case MOD_LR:
+ return is_restoration_used(cm) &&
+ (cm->rst_info[plane].frame_restoration_type != RESTORE_NONE);
+ default: assert(0);
+ }
+ return 0;
+}
+
/*!\brief Select and apply cdef filters and switchable restoration filters
*
* \ingroup high_level_algo
@@ -2135,16 +2210,19 @@
// Find CDEF parameters
av1_cdef_search(&cpi->mt_info, &cm->cur_frame->buf, cpi->source, cm, xd,
cpi->sf.lpf_sf.cdef_pick_method, cpi->td.mb.rdmult,
- cpi->sf.rt_sf.skip_cdef_sb, cpi->rc.frames_since_key,
- cpi->oxcf.tool_cfg.cdef_control, use_screen_content_model,
- cpi->svc.non_reference_frame);
+ cpi->sf.rt_sf.skip_cdef_sb, cpi->oxcf.tool_cfg.cdef_control,
+ use_screen_content_model, cpi->rtc_ref.non_reference_frame);
// Apply the filter
- if (!cpi->svc.non_reference_frame) {
+ if (!cpi->rtc_ref.non_reference_frame) {
if (num_workers > 1) {
+ // Extension of frame borders is multi-threaded along with cdef.
+ const int do_extend_border =
+ extend_borders_mt(cpi, MOD_CDEF, /* plane */ 0);
av1_cdef_frame_mt(cm, xd, cpi->mt_info.cdef_worker,
cpi->mt_info.workers, &cpi->mt_info.cdef_sync,
- num_workers, av1_cdef_init_fb_row_mt);
+ num_workers, av1_cdef_init_fb_row_mt,
+ do_extend_border);
} else {
av1_cdef_frame(&cm->cur_frame->buf, cm, xd, av1_cdef_init_fb_row);
}
@@ -2173,13 +2251,17 @@
if (cm->rst_info[0].frame_restoration_type != RESTORE_NONE ||
cm->rst_info[1].frame_restoration_type != RESTORE_NONE ||
cm->rst_info[2].frame_restoration_type != RESTORE_NONE) {
- if (num_workers > 1)
+ if (num_workers > 1) {
+ // Extension of frame borders is multi-threaded along with loop
+ // restoration filter.
+ const int do_extend_border = 1;
av1_loop_restoration_filter_frame_mt(
&cm->cur_frame->buf, cm, 0, mt_info->workers, num_workers,
- &mt_info->lr_row_sync, &cpi->lr_ctxt);
- else
+ &mt_info->lr_row_sync, &cpi->lr_ctxt, do_extend_border);
+ } else {
av1_loop_restoration_filter_frame(&cm->cur_frame->buf, cm, 0,
&cpi->lr_ctxt);
+ }
}
} else {
cm->rst_info[0].frame_restoration_type = RESTORE_NONE;
@@ -2208,8 +2290,7 @@
const int use_loopfilter =
!cm->features.coded_lossless && !cm->tiles.large_scale;
- const int use_cdef = cm->seq_params->enable_cdef &&
- !cm->features.coded_lossless && !cm->tiles.large_scale;
+ const int use_cdef = is_cdef_used(cm);
const int use_restoration = is_restoration_used(cm);
// lpf_opt_level = 1 : Enables dual/quad loop-filtering.
// lpf_opt_level is set to 1 if transform size search depth in inter blocks
@@ -2236,7 +2317,7 @@
}
if ((lf->filter_level[0] || lf->filter_level[1]) &&
- !cpi->svc.non_reference_frame) {
+ !cpi->rtc_ref.non_reference_frame) {
av1_loop_filter_frame_mt(&cm->cur_frame->buf, cm, xd, 0, num_planes, 0,
mt_info->workers, num_workers,
&mt_info->lf_row_sync, lpf_opt_level);
@@ -2448,8 +2529,8 @@
if (q_cfg->aq_mode == CYCLIC_REFRESH_AQ) {
suppress_active_map(cpi);
av1_cyclic_refresh_setup(cpi);
- av1_apply_active_map(cpi);
}
+ av1_apply_active_map(cpi);
if (cm->seg.enabled) {
if (!cm->seg.update_data && cm->prev_frame) {
segfeatures_copy(&cm->seg, &cm->prev_frame->seg);
@@ -3035,8 +3116,15 @@
}
// TODO(debargha): Fix mv search range on encoder side
- // aom_extend_frame_inner_borders(&cm->cur_frame->buf, av1_num_planes(cm));
- aom_extend_frame_borders(&cm->cur_frame->buf, av1_num_planes(cm));
+ for (int plane = 0; plane < av1_num_planes(cm); ++plane) {
+ const int extend_border_done = extend_borders_mt(cpi, MOD_CDEF, plane) ||
+ extend_borders_mt(cpi, MOD_LR, plane);
+ if (extend_border_done == 0) {
+ const YV12_BUFFER_CONFIG *ybf = &cm->cur_frame->buf;
+ aom_extend_frame_borders_plane_row(ybf, plane, 0,
+ ybf->crop_heights[plane > 0]);
+ }
+ }
#ifdef OUTPUT_YUV_REC
aom_write_one_yuv_frame(cm, &cm->cur_frame->buf);
@@ -3255,12 +3343,13 @@
if (cpi->svc.number_spatial_layers == 1 &&
cpi->svc.number_temporal_layers == 1) {
// Don't disable on intra_only, scene change (high_source_sad = 1),
- // or resized frame. To avoid quality loss for now, force enable at
- // every 8 frames.
+ // or resized frame. To avoid quality loss force enable at
+ // for ~30 frames after key or scene/slide change, and
+ // after 8 frames since last update if frame_source_sad > 0.
if (frame_is_intra_only(cm) || is_frame_resize_pending(cpi) ||
- rc->high_source_sad || rc->frames_since_key < 10 ||
- cpi->cyclic_refresh->counter_encode_maxq_scene_change < 10 ||
- cm->current_frame.frame_number % 8 == 0)
+ rc->high_source_sad || rc->frames_since_key < 30 ||
+ cpi->cyclic_refresh->counter_encode_maxq_scene_change < 30 ||
+ (cpi->frames_since_last_update > 8 && cpi->rc.frame_source_sad > 0))
return 0;
else
return 1;
@@ -3597,7 +3686,7 @@
break;
case 1: // Enable CDF update for all frames.
if (cpi->sf.rt_sf.disable_cdf_update_non_reference_frame &&
- cpi->svc.non_reference_frame && cpi->rc.frames_since_key > 2)
+ cpi->rtc_ref.non_reference_frame && cpi->rc.frames_since_key > 2)
features->disable_cdf_update = 1;
else
features->disable_cdf_update = 0;
@@ -3893,8 +3982,11 @@
#endif // CONFIG_DENOISE
if (av1_lookahead_push(cpi->ppi->lookahead, sd, time_stamp, end_time,
- use_highbitdepth, frame_flags))
+ use_highbitdepth, frame_flags)) {
+ aom_internal_error(cm->error, AOM_CODEC_ERROR,
+ "av1_lookahead_push() failed");
res = -1;
+ }
#if CONFIG_INTERNAL_STATS
aom_usec_timer_mark(&timer);
cpi->ppi->total_time_receive_data += aom_usec_timer_elapsed(&timer);
@@ -4210,7 +4302,7 @@
// We should fix the cpi->common.show_frame flag
// instead of checking the other condition to update the counter properly.
if (cpi->common.show_frame ||
- is_frame_droppable(&cpi->svc, &cpi->ext_flags.refresh_frame)) {
+ is_frame_droppable(&cpi->rtc_ref, &cpi->ext_flags.refresh_frame)) {
// Decrement count down till next gf
if (cpi->rc.frames_till_gf_update_due > 0)
cpi->rc.frames_till_gf_update_due--;
@@ -4895,10 +4987,11 @@
int av1_set_internal_size(AV1EncoderConfig *const oxcf,
ResizePendingParams *resize_pending_params,
- AOM_SCALING horiz_mode, AOM_SCALING vert_mode) {
+ AOM_SCALING_MODE horiz_mode,
+ AOM_SCALING_MODE vert_mode) {
int hr = 0, hs = 0, vr = 0, vs = 0;
- if (horiz_mode > ONETWO || vert_mode > ONETWO) return -1;
+ if (horiz_mode > AOME_ONETWO || vert_mode > AOME_ONETWO) return -1;
Scale2Ratio(horiz_mode, &hr, &hs);
Scale2Ratio(vert_mode, &vr, &vs);
@@ -4907,7 +5000,7 @@
resize_pending_params->width = (hs - 1 + oxcf->frm_dim_cfg.width * hr) / hs;
resize_pending_params->height = (vs - 1 + oxcf->frm_dim_cfg.height * vr) / vs;
- if (horiz_mode != NORMAL || vert_mode != NORMAL) {
+ if (horiz_mode != AOME_NORMAL || vert_mode != AOME_NORMAL) {
oxcf->resize_cfg.resize_mode = RESIZE_FIXED;
oxcf->algo_cfg.enable_tpl_model = 0;
}
@@ -4975,29 +5068,33 @@
return AOM_CODEC_OK;
}
-static void svc_set_updates_ref_frame_config(
- ExtRefreshFrameFlagsInfo *const ext_refresh_frame_flags, SVC *const svc) {
+static void rtc_set_updates_ref_frame_config(
+ ExtRefreshFrameFlagsInfo *const ext_refresh_frame_flags,
+ RTC_REF *const rtc_ref) {
ext_refresh_frame_flags->update_pending = 1;
- ext_refresh_frame_flags->last_frame = svc->refresh[svc->ref_idx[0]];
- ext_refresh_frame_flags->golden_frame = svc->refresh[svc->ref_idx[3]];
- ext_refresh_frame_flags->bwd_ref_frame = svc->refresh[svc->ref_idx[4]];
- ext_refresh_frame_flags->alt2_ref_frame = svc->refresh[svc->ref_idx[5]];
- ext_refresh_frame_flags->alt_ref_frame = svc->refresh[svc->ref_idx[6]];
- svc->non_reference_frame = 1;
+ ext_refresh_frame_flags->last_frame = rtc_ref->refresh[rtc_ref->ref_idx[0]];
+ ext_refresh_frame_flags->golden_frame = rtc_ref->refresh[rtc_ref->ref_idx[3]];
+ ext_refresh_frame_flags->bwd_ref_frame =
+ rtc_ref->refresh[rtc_ref->ref_idx[4]];
+ ext_refresh_frame_flags->alt2_ref_frame =
+ rtc_ref->refresh[rtc_ref->ref_idx[5]];
+ ext_refresh_frame_flags->alt_ref_frame =
+ rtc_ref->refresh[rtc_ref->ref_idx[6]];
+ rtc_ref->non_reference_frame = 1;
for (int i = 0; i < REF_FRAMES; i++) {
- if (svc->refresh[i] == 1) {
- svc->non_reference_frame = 0;
+ if (rtc_ref->refresh[i] == 1) {
+ rtc_ref->non_reference_frame = 0;
break;
}
}
}
-static int svc_set_references_external_ref_frame_config(AV1_COMP *cpi) {
+static int rtc_set_references_external_ref_frame_config(AV1_COMP *cpi) {
// LAST_FRAME (0), LAST2_FRAME(1), LAST3_FRAME(2), GOLDEN_FRAME(3),
// BWDREF_FRAME(4), ALTREF2_FRAME(5), ALTREF_FRAME(6).
int ref = AOM_REFFRAME_ALL;
for (int i = 0; i < INTER_REFS_PER_FRAME; i++) {
- if (!cpi->svc.reference[i]) ref ^= (1 << i);
+ if (!cpi->rtc_ref.reference[i]) ref ^= (1 << i);
}
return ref;
}
@@ -5036,8 +5133,8 @@
av1_use_as_reference(&ext_flags->ref_frame_flags, ref);
} else {
- if (cpi->svc.set_ref_frame_config) {
- int ref = svc_set_references_external_ref_frame_config(cpi);
+ if (cpi->rtc_ref.set_ref_frame_config) {
+ int ref = rtc_set_references_external_ref_frame_config(cpi);
av1_use_as_reference(&ext_flags->ref_frame_flags, ref);
}
}
@@ -5064,8 +5161,8 @@
ext_refresh_frame_flags->alt2_ref_frame = (upd & AOM_ALT2_FLAG) != 0;
ext_refresh_frame_flags->update_pending = 1;
} else {
- if (cpi->svc.set_ref_frame_config)
- svc_set_updates_ref_frame_config(ext_refresh_frame_flags, &cpi->svc);
+ if (cpi->rtc_ref.set_ref_frame_config)
+ rtc_set_updates_ref_frame_config(ext_refresh_frame_flags, &cpi->rtc_ref);
else
ext_refresh_frame_flags->update_pending = 0;
}
diff --git a/av1/encoder/encoder.h b/av1/encoder/encoder.h
index bd6c7a2..ca99531 100644
--- a/av1/encoder/encoder.h
+++ b/av1/encoder/encoder.h
@@ -104,16 +104,6 @@
} aom_rational64_t; // alias for struct aom_rational
enum {
- NORMAL = 0,
- FOURFIVE = 1,
- THREEFIVE = 2,
- THREEFOUR = 3,
- ONEFOUR = 4,
- ONEEIGHT = 5,
- ONETWO = 6
-} UENUM1BYTE(AOM_SCALING);
-
-enum {
// Good Quality Fast Encoding. The encoder balances quality with the amount of
// time it takes to encode the output. Speed setting controls how fast.
GOOD,
@@ -200,8 +190,6 @@
#define MAX_VBR_CORPUS_COMPLEXITY 10000
-/*!\cond */
-
typedef enum {
MOD_FP, // First pass
MOD_TF, // Temporal filtering
@@ -1067,6 +1055,9 @@
// Exit the encoder when it fails to encode to a given level.
int strict_level_conformance;
+
+ // Max depth for the GOP after a key frame
+ int kf_max_pyr_height;
/*!\endcond */
} AV1EncoderConfig;
@@ -1363,11 +1354,17 @@
#endif // CONFIG_MULTITHREAD
/*!
* Buffer to store the superblock whose encoding is complete.
- * cur_col[i] stores the number of superblocks which finished encoding in the
- * ith superblock row.
+ * num_finished_cols[i] stores the number of superblocks which finished
+ * encoding in the ith superblock row.
*/
int *num_finished_cols;
/*!
+ * Buffer to store the mi position of the block whose encoding is complete.
+ * finished_block_in_mi[i] stores the mi position of the block which finished
+ * encoding in the ith superblock row.
+ */
+ int *finished_block_in_mi;
+ /*!
* Denotes the superblock interval at which conditional signalling should
* happen. Also denotes the minimum number of extra superblocks of the top row
* to be complete to start encoding the current superblock. A value of 1
@@ -2354,6 +2351,22 @@
/*!\endcond */
#endif
+/*!\cond */
+typedef struct RTC_REF {
+ /*!
+ * LAST_FRAME (0), LAST2_FRAME(1), LAST3_FRAME(2), GOLDEN_FRAME(3),
+ * BWDREF_FRAME(4), ALTREF2_FRAME(5), ALTREF_FRAME(6).
+ */
+ int reference[INTER_REFS_PER_FRAME];
+ int ref_idx[INTER_REFS_PER_FRAME];
+ int refresh[REF_FRAMES];
+ int set_ref_frame_config;
+ int non_reference_frame;
+ int ref_frame_comp[3];
+ int gld_idx_1layer;
+} RTC_REF;
+/*!\endcond */
+
/*!
* \brief Structure to hold data corresponding to an encoded frame.
*/
@@ -3393,6 +3406,16 @@
* Frames since last frame with cdf update.
*/
int frames_since_last_update;
+
+ /*!
+ * Struct for the reference structure for RTC.
+ */
+ RTC_REF rtc_ref;
+
+ /*!
+ * Block level thresholds to force zeromv-skip at partition level.
+ */
+ unsigned int zeromv_skip_thresh_exit_part[BLOCK_SIZES_ALL];
} AV1_COMP;
/*!
@@ -3604,7 +3627,8 @@
int av1_set_internal_size(AV1EncoderConfig *const oxcf,
ResizePendingParams *resize_pending_params,
- AOM_SCALING horiz_mode, AOM_SCALING vert_mode);
+ AOM_SCALING_MODE horiz_mode,
+ AOM_SCALING_MODE vert_mode);
int av1_get_quantizer(struct AV1_COMP *cpi);
@@ -3739,8 +3763,10 @@
// the frame token allocation.
static INLINE unsigned int allocated_tokens(const TileInfo *tile,
int sb_size_log2, int num_planes) {
- int tile_mb_rows = (tile->mi_row_end - tile->mi_row_start + 2) >> 2;
- int tile_mb_cols = (tile->mi_col_end - tile->mi_col_start + 2) >> 2;
+ int tile_mb_rows =
+ ROUND_POWER_OF_TWO(tile->mi_row_end - tile->mi_row_start, 2);
+ int tile_mb_cols =
+ ROUND_POWER_OF_TWO(tile->mi_col_end - tile->mi_col_start, 2);
return get_token_alloc(tile_mb_rows, tile_mb_cols, sb_size_log2, num_planes);
}
@@ -3825,9 +3851,11 @@
cpi->oxcf.gf_cfg.lag_in_frames == 0;
}
-static INLINE int use_one_pass_rt_reference_structure(const AV1_COMP *cpi) {
- return cpi->oxcf.speed >= 5 && cpi->ppi->number_spatial_layers == 1 &&
- cpi->ppi->number_temporal_layers == 1;
+// Use default/internal reference structure for single-layer RTC.
+static INLINE int use_rtc_reference_structure_one_layer(const AV1_COMP *cpi) {
+ return is_one_pass_rt_params(cpi) && cpi->ppi->number_spatial_layers == 1 &&
+ cpi->ppi->number_temporal_layers == 1 &&
+ !cpi->rtc_ref.set_ref_frame_config;
}
// Function return size of frame stats buffer
@@ -4046,6 +4074,12 @@
cpi->common.height != resize_pending_params->height));
}
+// Check if CDEF is used.
+static INLINE int is_cdef_used(const AV1_COMMON *const cm) {
+ return cm->seq_params->enable_cdef && !cm->features.coded_lossless &&
+ !cm->tiles.large_scale;
+}
+
// Check if loop restoration filter is used.
static INLINE int is_restoration_used(const AV1_COMMON *const cm) {
return cm->seq_params->enable_restoration && !cm->features.all_lossless &&
@@ -4058,6 +4092,12 @@
tx_sf->inter_tx_size_search_init_depth_sqr >= 1);
}
+// Enable switchable motion mode only if warp and OBMC tools are allowed
+static INLINE bool is_switchable_motion_mode_allowed(bool allow_warped_motion,
+ bool enable_obmc) {
+ return (allow_warped_motion || enable_obmc);
+}
+
#if CONFIG_AV1_TEMPORAL_DENOISING
static INLINE int denoise_svc(const struct AV1_COMP *const cpi) {
return (!cpi->ppi->use_svc ||
diff --git a/av1/encoder/encoder_alloc.h b/av1/encoder/encoder_alloc.h
index 72c823e..fd350f0 100644
--- a/av1/encoder/encoder_alloc.h
+++ b/av1/encoder/encoder_alloc.h
@@ -294,6 +294,8 @@
}
if (cpi->ppi->use_svc) av1_free_svc_cyclic_refresh(cpi);
+ aom_free(cpi->svc.layer_context);
+ cpi->svc.layer_context = NULL;
if (cpi->consec_zero_mv) {
aom_free(cpi->consec_zero_mv);
diff --git a/av1/encoder/encoder_utils.c b/av1/encoder/encoder_utils.c
index 75dfd9b..0c69e4d 100644
--- a/av1/encoder/encoder_utils.c
+++ b/av1/encoder/encoder_utils.c
@@ -636,7 +636,6 @@
void av1_update_film_grain_parameters(struct AV1_COMP *cpi,
const AV1EncoderConfig *oxcf) {
AV1_COMMON *const cm = &cpi->common;
- cpi->oxcf = *oxcf;
const TuneCfg *const tune_cfg = &oxcf->tune_cfg;
if (cpi->film_grain_table) {
@@ -796,7 +795,10 @@
? BLOCK_128X128
: BLOCK_64X64;
} else if (oxcf->mode == REALTIME) {
- return AOMMIN(width, height) > 720 ? BLOCK_128X128 : BLOCK_64X64;
+ if (oxcf->tune_cfg.content == AOM_CONTENT_SCREEN)
+ return AOMMIN(width, height) >= 720 ? BLOCK_128X128 : BLOCK_64X64;
+ else
+ return AOMMIN(width, height) > 720 ? BLOCK_128X128 : BLOCK_64X64;
}
// TODO(any): Possibly could improve this with a heuristic.
diff --git a/av1/encoder/encoder_utils.h b/av1/encoder/encoder_utils.h
index 44294db..dd91bb1 100644
--- a/av1/encoder/encoder_utils.h
+++ b/av1/encoder/encoder_utils.h
@@ -49,22 +49,26 @@
seg_map[i] = AM_SEGMENT_ID_ACTIVE;
}
-static AOM_INLINE void set_mb_mi(CommonModeInfoParams *mi_params, int width,
- int height) {
+// Returns 'size' in the number of Mode Info (MI) units. 'size' is either the
+// width or height.
+static AOM_INLINE int size_in_mi(int size) {
// Ensure that the decoded width and height are both multiples of
// 8 luma pixels (note: this may only be a multiple of 4 chroma pixels if
// subsampling is used).
// This simplifies the implementation of various experiments,
// eg. cdef, which operates on units of 8x8 luma pixels.
- const int aligned_width = ALIGN_POWER_OF_TWO(width, 3);
- const int aligned_height = ALIGN_POWER_OF_TWO(height, 3);
+ const int aligned_size = ALIGN_POWER_OF_TWO(size, 3);
+ return aligned_size >> MI_SIZE_LOG2;
+}
- mi_params->mi_cols = aligned_width >> MI_SIZE_LOG2;
- mi_params->mi_rows = aligned_height >> MI_SIZE_LOG2;
+static AOM_INLINE void set_mb_mi(CommonModeInfoParams *mi_params, int width,
+ int height) {
+ mi_params->mi_cols = size_in_mi(width);
+ mi_params->mi_rows = size_in_mi(height);
mi_params->mi_stride = calc_mi_size(mi_params->mi_cols);
- mi_params->mb_cols = (mi_params->mi_cols + 2) >> 2;
- mi_params->mb_rows = (mi_params->mi_rows + 2) >> 2;
+ mi_params->mb_cols = ROUND_POWER_OF_TWO(mi_params->mi_cols, 2);
+ mi_params->mb_rows = ROUND_POWER_OF_TWO(mi_params->mi_rows, 2);
mi_params->MBs = mi_params->mb_rows * mi_params->mb_cols;
const int mi_alloc_size_1d = mi_size_wide[mi_params->mi_alloc_bsize];
@@ -970,6 +974,7 @@
static AOM_INLINE void set_size_independent_vars(AV1_COMP *cpi) {
int i;
AV1_COMMON *const cm = &cpi->common;
+ FeatureFlags *const features = &cm->features;
for (i = LAST_FRAME; i <= ALTREF_FRAME; ++i) {
cm->global_motion[i] = default_warp_params;
}
@@ -977,8 +982,9 @@
av1_set_speed_features_framesize_independent(cpi, cpi->speed);
av1_set_rd_speed_thresholds(cpi);
- cm->features.interp_filter = SWITCHABLE;
- cm->features.switchable_motion_mode = 1;
+ features->interp_filter = SWITCHABLE;
+ features->switchable_motion_mode = is_switchable_motion_mode_allowed(
+ features->allow_warped_motion, cpi->oxcf.motion_mode_cfg.enable_obmc);
}
static AOM_INLINE void release_scaled_references(AV1_COMP *cpi) {
@@ -1000,7 +1006,7 @@
static AOM_INLINE int reduce_num_ref_buffers(const AV1_COMP *cpi) {
const SequenceHeader *const seq_params = cpi->common.seq_params;
return is_one_pass_rt_params(cpi) &&
- use_one_pass_rt_reference_structure(cpi) &&
+ use_rtc_reference_structure_one_layer(cpi) &&
(seq_params->order_hint_info.enable_order_hint == 0) &&
cpi->rt_reduce_num_ref_buffers;
}
diff --git a/av1/encoder/ethread.c b/av1/encoder/ethread.c
index 7be7768..3ef6b17 100644
--- a/av1/encoder/ethread.c
+++ b/av1/encoder/ethread.c
@@ -187,6 +187,9 @@
CHECK_MEM_ERROR(cm, row_mt_sync->num_finished_cols,
aom_malloc(sizeof(*row_mt_sync->num_finished_cols) * rows));
+ CHECK_MEM_ERROR(
+ cm, row_mt_sync->finished_block_in_mi,
+ aom_malloc(sizeof(*row_mt_sync->finished_block_in_mi) * rows));
row_mt_sync->rows = rows;
// Set up nsync.
@@ -213,6 +216,7 @@
}
#endif // CONFIG_MULTITHREAD
aom_free(row_mt_sync->num_finished_cols);
+ aom_free(row_mt_sync->finished_block_in_mi);
// clear the structure as the source of this call may be dynamic change
// in tiles in which case this call will be followed by an _alloc()
@@ -444,6 +448,12 @@
pthread_mutex_t *enc_row_mt_mutex_ = enc_row_mt->mutex_;
#endif
(void)unused;
+ // Preallocate the pc_tree for realtime coding to reduce the cost of memory
+ // allocation.
+ thread_data->td->rt_pc_root =
+ cpi->sf.rt_sf.use_nonrd_pick_mode
+ ? av1_alloc_pc_tree_node(cm->seq_params->sb_size)
+ : NULL;
assert(cur_tile_id != -1);
@@ -515,6 +525,8 @@
#endif
}
+ av1_free_pc_tree_recursive(thread_data->td->rt_pc_root, av1_num_planes(cm), 0,
+ 0);
return 1;
}
@@ -527,6 +539,12 @@
int t;
(void)unused;
+ // Preallocate the pc_tree for realtime coding to reduce the cost of memory
+ // allocation.
+ thread_data->td->rt_pc_root =
+ cpi->sf.rt_sf.use_nonrd_pick_mode
+ ? av1_alloc_pc_tree_node(cm->seq_params->sb_size)
+ : NULL;
for (t = thread_data->start; t < tile_rows * tile_cols;
t += cpi->mt_info.num_workers) {
@@ -540,6 +558,9 @@
av1_encode_tile(cpi, thread_data->td, tile_row, tile_col);
}
+ av1_free_pc_tree_recursive(thread_data->td->rt_pc_root, av1_num_planes(cm), 0,
+ 0);
+
return 1;
}
@@ -1221,9 +1242,6 @@
thread_data->td->mb.txfm_search_info.tx_search_count;
#endif // CONFIG_SPEED_STATS
}
-
- av1_free_pc_tree_recursive(thread_data->td->rt_pc_root,
- av1_num_planes(&cpi->common), 0, 0);
}
}
@@ -1322,13 +1340,6 @@
thread_data->td->mb.tmp_pred_bufs[j];
}
}
-
- // Preallocate the pc_tree for realtime coding to reduce the cost of memory
- // allocation.
- thread_data->td->rt_pc_root =
- cpi->sf.rt_sf.use_nonrd_pick_mode
- ? av1_alloc_pc_tree_node(cm->seq_params->sb_size)
- : NULL;
}
}
@@ -1559,6 +1570,8 @@
// Initialize num_finished_cols to -1 for all rows.
memset(row_mt_sync->num_finished_cols, -1,
sizeof(*row_mt_sync->num_finished_cols) * max_sb_rows);
+ memset(row_mt_sync->finished_block_in_mi, -1,
+ sizeof(*row_mt_sync->finished_block_in_mi) * max_sb_rows);
row_mt_sync->next_mi_row = this_tile->tile_info.mi_row_start;
row_mt_sync->num_threads_working = 0;
row_mt_sync->intrabc_extra_top_right_sb_delay =
diff --git a/av1/encoder/firstpass.c b/av1/encoder/firstpass.c
index 7ad0c8d..8434208 100644
--- a/av1/encoder/firstpass.c
+++ b/av1/encoder/firstpass.c
@@ -36,6 +36,7 @@
#include "av1/encoder/encodemb.h"
#include "av1/encoder/encodemv.h"
#include "av1/encoder/encoder.h"
+#include "av1/encoder/encoder_utils.h"
#include "av1/encoder/encode_strategy.h"
#include "av1/encoder/ethread.h"
#include "av1/encoder/extend.h"
@@ -473,8 +474,10 @@
set_mi_offsets(mi_params, xd, unit_row * unit_scale, unit_col * unit_scale);
xd->plane[0].dst.buf = this_frame->y_buffer + y_offset;
- xd->plane[1].dst.buf = this_frame->u_buffer + uv_offset;
- xd->plane[2].dst.buf = this_frame->v_buffer + uv_offset;
+ if (num_planes > 1) {
+ xd->plane[1].dst.buf = this_frame->u_buffer + uv_offset;
+ xd->plane[2].dst.buf = this_frame->v_buffer + uv_offset;
+ }
xd->left_available = (unit_col != 0);
xd->mi[0]->bsize = bsize;
xd->mi[0]->ref_frame[0] = INTRA_FRAME;
@@ -761,8 +764,10 @@
// Reset to last frame as reference buffer.
xd->plane[0].pre[0].buf = last_frame->y_buffer + recon_yoffset;
- xd->plane[1].pre[0].buf = last_frame->u_buffer + recon_uvoffset;
- xd->plane[2].pre[0].buf = last_frame->v_buffer + recon_uvoffset;
+ if (av1_num_planes(&cpi->common) > 1) {
+ xd->plane[1].pre[0].buf = last_frame->u_buffer + recon_uvoffset;
+ xd->plane[2].pre[0].buf = last_frame->v_buffer + recon_uvoffset;
+ }
} else {
stats->sr_coded_error += motion_error;
}
@@ -1196,8 +1201,10 @@
// Adjust to the next column of MBs.
x->plane[0].src.buf += fp_block_size_width;
- x->plane[1].src.buf += uv_mb_height;
- x->plane[2].src.buf += uv_mb_height;
+ if (num_planes > 1) {
+ x->plane[1].src.buf += uv_mb_height;
+ x->plane[2].src.buf += uv_mb_height;
+ }
recon_yoffset += fp_block_size_width;
src_yoffset += fp_block_size_width;
@@ -1213,8 +1220,18 @@
AV1_COMMON *const cm = &cpi->common;
CurrentFrame *const current_frame = &cm->current_frame;
const CommonModeInfoParams *const mi_params = &cm->mi_params;
- const int unit_rows = get_unit_rows(BLOCK_16X16, mi_params->mb_rows);
- const int unit_cols = get_unit_cols(BLOCK_16X16, mi_params->mb_cols);
+ int max_mb_rows = mi_params->mb_rows;
+ int max_mb_cols = mi_params->mb_cols;
+ if (cpi->oxcf.frm_dim_cfg.forced_max_frame_width) {
+ int max_mi_cols = size_in_mi(cpi->oxcf.frm_dim_cfg.forced_max_frame_width);
+ max_mb_cols = ROUND_POWER_OF_TWO(max_mi_cols, 2);
+ }
+ if (cpi->oxcf.frm_dim_cfg.forced_max_frame_height) {
+ int max_mi_rows = size_in_mi(cpi->oxcf.frm_dim_cfg.forced_max_frame_height);
+ max_mb_rows = ROUND_POWER_OF_TWO(max_mi_rows, 2);
+ }
+ const int unit_rows = get_unit_rows(BLOCK_16X16, max_mb_rows);
+ const int unit_cols = get_unit_cols(BLOCK_16X16, max_mb_cols);
setup_firstpass_data(cm, &cpi->firstpass_data, unit_rows, unit_cols);
FRAME_STATS *mb_stats = cpi->firstpass_data.mb_stats;
FRAME_STATS stats = accumulate_frame_stats(mb_stats, unit_rows, unit_cols);
@@ -1248,10 +1265,21 @@
const BLOCK_SIZE fp_block_size =
get_fp_block_size(cpi->is_screen_content_type);
+ int max_mb_rows = mi_params->mb_rows;
+ int max_mb_cols = mi_params->mb_cols;
+ if (cpi->oxcf.frm_dim_cfg.forced_max_frame_width) {
+ int max_mi_cols = size_in_mi(cpi->oxcf.frm_dim_cfg.forced_max_frame_width);
+ max_mb_cols = ROUND_POWER_OF_TWO(max_mi_cols, 2);
+ }
+ if (cpi->oxcf.frm_dim_cfg.forced_max_frame_height) {
+ int max_mi_rows = size_in_mi(cpi->oxcf.frm_dim_cfg.forced_max_frame_height);
+ max_mb_rows = ROUND_POWER_OF_TWO(max_mi_rows, 2);
+ }
+
// Number of rows in the unit size.
- // Note mi_params->mb_rows and mi_params->mb_cols are in the unit of 16x16.
- const int unit_rows = get_unit_rows(fp_block_size, mi_params->mb_rows);
- const int unit_cols = get_unit_cols(fp_block_size, mi_params->mb_cols);
+ // Note max_mb_rows and max_mb_cols are in the unit of 16x16.
+ const int unit_rows = get_unit_rows(fp_block_size, max_mb_rows);
+ const int unit_cols = get_unit_cols(fp_block_size, max_mb_cols);
// Set fp_block_size, for the convenience of multi-thread usage.
cpi->fp_block_size = fp_block_size;
diff --git a/av1/encoder/firstpass.h b/av1/encoder/firstpass.h
index ca4401b..be52e8e 100644
--- a/av1/encoder/firstpass.h
+++ b/av1/encoder/firstpass.h
@@ -578,7 +578,7 @@
* \param[in] cpi Top-level encoder structure
* \param[in] ts_duration Duration of the frame / collection of frames
*
- * \return Nothing is returned. Instead, the "TWO_PASS" structure inside "cpi"
+ * \remark Nothing is returned. Instead, the "TWO_PASS" structure inside "cpi"
* is modified to store information computed in this function.
*/
void av1_first_pass(struct AV1_COMP *cpi, const int64_t ts_duration);
diff --git a/av1/encoder/gop_structure.c b/av1/encoder/gop_structure.c
index ab20083..f1689de 100644
--- a/av1/encoder/gop_structure.c
+++ b/av1/encoder/gop_structure.c
@@ -705,8 +705,9 @@
}
} else {
// Set layer depth threshold for reordering as per the gf length.
- int depth_thr =
- (actual_gf_length == 16) ? 3 : (actual_gf_length == 32) ? 4 : INT_MAX;
+ int depth_thr = (actual_gf_length == 16) ? 3
+ : (actual_gf_length == 32) ? 4
+ : INT_MAX;
set_multi_layer_params_for_fp(
twopass, &cpi->twopass_frame, gf_group, p_rc, rc, frame_info,
@@ -824,10 +825,15 @@
const int key_frame = rc->frames_since_key == 0;
FRAME_UPDATE_TYPE first_frame_update_type = ARF_UPDATE;
- if (key_frame)
+ if (key_frame) {
first_frame_update_type = KF_UPDATE;
- else if (!cpi->ppi->gf_state.arf_gf_boost_lst)
+ if (cpi->oxcf.kf_max_pyr_height != -1) {
+ gf_group->max_layer_depth_allowed = AOMMIN(
+ cpi->oxcf.kf_max_pyr_height, gf_group->max_layer_depth_allowed);
+ }
+ } else if (!cpi->ppi->gf_state.arf_gf_boost_lst) {
first_frame_update_type = GF_UPDATE;
+ }
gf_group->size = construct_multi_layer_gf_structure(
cpi, twopass, gf_group, rc, frame_info, p_rc->baseline_gf_interval,
diff --git a/av1/encoder/gop_structure.h b/av1/encoder/gop_structure.h
index eb20c84..ff22f54 100644
--- a/av1/encoder/gop_structure.h
+++ b/av1/encoder/gop_structure.h
@@ -38,7 +38,7 @@
*
* \param[in] cpi Top - level encoder instance structure
*
- * \return No return value but this function updates group data structures.
+ * \remark No return value but this function updates group data structures.
*/
void av1_gop_setup_structure(struct AV1_COMP *cpi);
@@ -58,7 +58,7 @@
* uni-directional group.
* \param[in] gf_group_bits Bits available to be allocated.
*
- * \return No return but updates the rate control and group data structures
+ * \remark No return but updates the rate control and group data structures
* to reflect the allocation of bits.
*/
void av1_gop_bit_allocation(const AV1_COMP *cpi, RATE_CONTROL *const rc,
diff --git a/av1/encoder/intra_mode_search.c b/av1/encoder/intra_mode_search.c
index ab8e3c3..d863910 100644
--- a/av1/encoder/intra_mode_search.c
+++ b/av1/encoder/intra_mode_search.c
@@ -228,7 +228,7 @@
*/
static int rd_pick_filter_intra_sby(const AV1_COMP *const cpi, MACROBLOCK *x,
int *rate, int *rate_tokenonly,
- int64_t *distortion, int *skippable,
+ int64_t *distortion, uint8_t *skippable,
BLOCK_SIZE bsize, int mode_cost,
PREDICTION_MODE best_mode_so_far,
int64_t *best_rd, int64_t *best_model_rd,
@@ -812,7 +812,7 @@
int64_t av1_rd_pick_intra_sbuv_mode(const AV1_COMP *const cpi, MACROBLOCK *x,
int *rate, int *rate_tokenonly,
- int64_t *distortion, int *skippable,
+ int64_t *distortion, uint8_t *skippable,
BLOCK_SIZE bsize, TX_SIZE max_tx_size) {
const AV1_COMMON *const cm = &cpi->common;
MACROBLOCKD *xd = &x->e_mbd;
@@ -1137,7 +1137,8 @@
BLOCK_SIZE bsize, const int *bmode_costs,
int64_t *best_rd, int *rate,
int *rate_tokenonly, int64_t *distortion,
- int *skippable, MB_MODE_INFO *best_mbmi,
+ uint8_t *skippable,
+ MB_MODE_INFO *best_mbmi,
PICK_MODE_CONTEXT *ctx) {
MACROBLOCKD *const xd = &x->e_mbd;
MB_MODE_INFO *const mbmi = xd->mi[0];
@@ -1179,7 +1180,7 @@
* \callergraph
* This function loops through all filter_intra modes to find the best one.
*
- * \return Returns nothing, but updates the mbmi and rd_stats.
+ * \remark Returns nothing, but updates the mbmi and rd_stats.
*/
static INLINE void handle_filter_intra_mode(const AV1_COMP *cpi, MACROBLOCK *x,
BLOCK_SIZE bsize,
@@ -1431,7 +1432,7 @@
// Finds the best non-intrabc mode on an intra frame.
int64_t av1_rd_pick_intra_sby_mode(const AV1_COMP *const cpi, MACROBLOCK *x,
int *rate, int *rate_tokenonly,
- int64_t *distortion, int *skippable,
+ int64_t *distortion, uint8_t *skippable,
BLOCK_SIZE bsize, int64_t best_rd,
PICK_MODE_CONTEXT *ctx) {
MACROBLOCKD *const xd = &x->e_mbd;
diff --git a/av1/encoder/intra_mode_search.h b/av1/encoder/intra_mode_search.h
index 0a6b769..75289c4 100644
--- a/av1/encoder/intra_mode_search.h
+++ b/av1/encoder/intra_mode_search.h
@@ -62,7 +62,7 @@
int rate_uv_intra; /*!< \brief Total rate to transmit uv_mode */
int rate_uv_tokenonly; /*!< \brief Rate transmit txfm tokens */
int64_t dist_uvs; /*!< \brief Distortion of the uv_mode's recon */
- int skip_uvs; /*!< \brief Whether the uv txfm is skippable */
+ uint8_t skip_uvs; /*!< \brief Whether the uv txfm is skippable */
UV_PREDICTION_MODE mode_uv; /*!< \brief The best uv mode */
PALETTE_MODE_INFO pmi_uv; /*!< \brief Color map if mode_uv is palette */
int8_t uv_angle_delta; /*!< \brief Angle delta if mode_uv directional */
@@ -196,8 +196,6 @@
* \param[in] this_rd_cost Struct to keep track of palette mode's
* rd_stats.
* \param[in] best_rd Best RD seen for this block so far.
- *
- * \return Returns nothing.
*/
void av1_search_palette_mode_luma(const AV1_COMP *cpi, MACROBLOCK *x,
BLOCK_SIZE bsize, unsigned int ref_frame_cost,
@@ -236,7 +234,7 @@
*/
int64_t av1_rd_pick_intra_sby_mode(const AV1_COMP *const cpi, MACROBLOCK *x,
int *rate, int *rate_tokenonly,
- int64_t *distortion, int *skippable,
+ int64_t *distortion, uint8_t *skippable,
BLOCK_SIZE bsize, int64_t best_rd,
PICK_MODE_CONTEXT *ctx);
@@ -271,7 +269,7 @@
*/
int64_t av1_rd_pick_intra_sbuv_mode(const AV1_COMP *const cpi, MACROBLOCK *x,
int *rate, int *rate_tokenonly,
- int64_t *distortion, int *skippable,
+ int64_t *distortion, uint8_t *skippable,
BLOCK_SIZE bsize, TX_SIZE max_tx_size);
/*! \brief Return the number of colors in src. Used by palette mode.
diff --git a/av1/encoder/lookahead.c b/av1/encoder/lookahead.c
index a9bccb1..10fbb77 100644
--- a/av1/encoder/lookahead.c
+++ b/av1/encoder/lookahead.c
@@ -155,7 +155,10 @@
buf->flags = flags;
++ctx->push_frame_count;
aom_remove_metadata_from_frame_buffer(&buf->img);
- aom_copy_metadata_to_frame_buffer(&buf->img, src->metadata);
+ if (src->metadata &&
+ aom_copy_metadata_to_frame_buffer(&buf->img, src->metadata)) {
+ return 1;
+ }
return 0;
}
diff --git a/av1/encoder/model_rd.h b/av1/encoder/model_rd.h
index db5ede4..f7e8b96 100644
--- a/av1/encoder/model_rd.h
+++ b/av1/encoder/model_rd.h
@@ -35,13 +35,11 @@
#define MODELRD_TYPE_INTRA 1
#define MODELRD_TYPE_MOTION_MODE_RD 1
-typedef void (*model_rd_for_sb_type)(const AV1_COMP *const cpi,
- BLOCK_SIZE bsize, MACROBLOCK *x,
- MACROBLOCKD *xd, int plane_from,
- int plane_to, int *out_rate_sum,
- int64_t *out_dist_sum, int *skip_txfm_sb,
- int64_t *skip_sse_sb, int *plane_rate,
- int64_t *plane_sse, int64_t *plane_dist);
+typedef void (*model_rd_for_sb_type)(
+ const AV1_COMP *const cpi, BLOCK_SIZE bsize, MACROBLOCK *x, MACROBLOCKD *xd,
+ int plane_from, int plane_to, int *out_rate_sum, int64_t *out_dist_sum,
+ uint8_t *skip_txfm_sb, int64_t *skip_sse_sb, int *plane_rate,
+ int64_t *plane_sse, int64_t *plane_dist);
typedef void (*model_rd_from_sse_type)(const AV1_COMP *const cpi,
const MACROBLOCK *const x,
BLOCK_SIZE plane_bsize, int plane,
@@ -160,7 +158,7 @@
static AOM_INLINE void model_rd_for_sb(
const AV1_COMP *const cpi, BLOCK_SIZE bsize, MACROBLOCK *x, MACROBLOCKD *xd,
int plane_from, int plane_to, int *out_rate_sum, int64_t *out_dist_sum,
- int *skip_txfm_sb, int64_t *skip_sse_sb, int *plane_rate,
+ uint8_t *skip_txfm_sb, int64_t *skip_sse_sb, int *plane_rate,
int64_t *plane_sse, int64_t *plane_dist) {
// Note our transform coeffs are 8 times an orthogonal transform.
// Hence quantizer step is also 8 times. To get effective quantizer
@@ -212,7 +210,7 @@
static AOM_INLINE void model_rd_for_sb_with_curvfit(
const AV1_COMP *const cpi, BLOCK_SIZE bsize, MACROBLOCK *x, MACROBLOCKD *xd,
int plane_from, int plane_to, int *out_rate_sum, int64_t *out_dist_sum,
- int *skip_txfm_sb, int64_t *skip_sse_sb, int *plane_rate,
+ uint8_t *skip_txfm_sb, int64_t *skip_sse_sb, int *plane_rate,
int64_t *plane_sse, int64_t *plane_dist) {
// Note our transform coeffs are 8 times an orthogonal transform.
// Hence quantizer step is also 8 times. To get effective quantizer
diff --git a/av1/encoder/nonrd_opt.h b/av1/encoder/nonrd_opt.h
index 39049e5..8a25061 100644
--- a/av1/encoder/nonrd_opt.h
+++ b/av1/encoder/nonrd_opt.h
@@ -14,6 +14,27 @@
#include "av1/encoder/rdopt_utils.h"
+#define RTC_INTER_MODES (4)
+#define RTC_INTRA_MODES (4)
+#define RTC_MODES (AOMMAX(RTC_INTER_MODES, RTC_INTRA_MODES))
+
+static const PREDICTION_MODE intra_mode_list[] = { DC_PRED, V_PRED, H_PRED,
+ SMOOTH_PRED };
+
+static const PREDICTION_MODE inter_mode_list[] = { NEARESTMV, NEARMV, GLOBALMV,
+ NEWMV };
+
+static const THR_MODES mode_idx[REF_FRAMES][RTC_MODES] = {
+ { THR_DC, THR_V_PRED, THR_H_PRED, THR_SMOOTH },
+ { THR_NEARESTMV, THR_NEARMV, THR_GLOBALMV, THR_NEWMV },
+ { THR_NEARESTL2, THR_NEARL2, THR_GLOBALL2, THR_NEWL2 },
+ { THR_NEARESTL3, THR_NEARL3, THR_GLOBALL3, THR_NEWL3 },
+ { THR_NEARESTG, THR_NEARG, THR_GLOBALG, THR_NEWG },
+ { THR_NEARESTB, THR_NEARB, THR_GLOBALB, THR_NEWB },
+ { THR_NEARESTA2, THR_NEARA2, THR_GLOBALA2, THR_NEWA2 },
+ { THR_NEARESTA, THR_NEARA, THR_GLOBALA, THR_NEWA },
+};
+
/*!\brief Finds predicted motion vectors for a block.
*
* \ingroup nonrd_mode_search
@@ -37,7 +58,7 @@
* prune for low temporal variance block
* \param[in] skip_pred_mv Flag indicating to skip av1_mv_pred
*
- * \return Nothing is returned. Instead, predicted MVs are placed into
+ * \remark Nothing is returned. Instead, predicted MVs are placed into
* \c frame_mv array
*/
static INLINE void find_predictors(
@@ -80,7 +101,9 @@
bsize);
}
}
- av1_count_overlappable_neighbors(cm, xd);
+ if (cm->features.switchable_motion_mode) {
+ av1_count_overlappable_neighbors(cm, xd);
+ }
mbmi->num_proj_ref = 1;
}
diff --git a/av1/encoder/nonrd_pickmode.c b/av1/encoder/nonrd_pickmode.c
index 58fc07a..f85f2f0 100644
--- a/av1/encoder/nonrd_pickmode.c
+++ b/av1/encoder/nonrd_pickmode.c
@@ -38,6 +38,7 @@
#include "av1/encoder/reconinter_enc.h"
#include "av1/encoder/var_based_part.h"
+#define CALC_BIASED_RDCOST(rdcost) (7 * (rdcost) >> 3)
extern int g_pick_inter_mode_cnt;
/*!\cond */
typedef struct {
@@ -82,10 +83,7 @@
#define NUM_INTER_MODES_RT 9
#define NUM_COMP_INTER_MODES_RT (6)
-#define NUM_INTER_MODES_REDUCED 8
-#define RTC_INTER_MODES (4)
-#define RTC_INTRA_MODES (4)
-#define RTC_MODES (AOMMAX(RTC_INTER_MODES, RTC_INTRA_MODES))
+#define NUM_INTER_MODES_REDUCED 12
static const REF_MODE ref_mode_set_rt[NUM_INTER_MODES_RT] = {
{ LAST_FRAME, NEARESTMV }, { LAST_FRAME, NEARMV },
@@ -102,17 +100,8 @@
{ LAST_FRAME, GLOBALMV }, { LAST_FRAME, NEWMV },
{ GOLDEN_FRAME, NEARESTMV }, { GOLDEN_FRAME, NEARMV },
{ GOLDEN_FRAME, GLOBALMV }, { GOLDEN_FRAME, NEWMV },
-};
-
-static const THR_MODES mode_idx[REF_FRAMES][RTC_MODES] = {
- { THR_DC, THR_V_PRED, THR_H_PRED, THR_SMOOTH },
- { THR_NEARESTMV, THR_NEARMV, THR_GLOBALMV, THR_NEWMV },
- { THR_NEARESTL2, THR_NEARL2, THR_GLOBALL2, THR_NEWL2 },
- { THR_NEARESTL3, THR_NEARL3, THR_GLOBALL3, THR_NEWL3 },
- { THR_NEARESTG, THR_NEARG, THR_GLOBALG, THR_NEWG },
- { THR_NEARESTB, THR_NEARB, THR_GLOBALB, THR_NEWB },
- { THR_NEARESTA2, THR_NEARA2, THR_GLOBALA2, THR_NEWA2 },
- { THR_NEARESTA, THR_NEARA, THR_GLOBALA, THR_NEWA },
+ { ALTREF_FRAME, NEARESTMV }, { ALTREF_FRAME, NEARMV },
+ { ALTREF_FRAME, GLOBALMV }, { ALTREF_FRAME, NEWMV },
};
static const COMP_REF_MODE comp_ref_mode_set[NUM_COMP_INTER_MODES_RT] = {
@@ -124,9 +113,6 @@
{ { LAST_FRAME, ALTREF_FRAME }, NEAREST_NEARESTMV },
};
-static const PREDICTION_MODE intra_mode_list[] = { DC_PRED, V_PRED, H_PRED,
- SMOOTH_PRED };
-
static const INTER_FILTER filters_ref_set[9] = {
{ EIGHTTAP_REGULAR, EIGHTTAP_REGULAR }, { EIGHTTAP_SMOOTH, EIGHTTAP_SMOOTH },
{ EIGHTTAP_REGULAR, EIGHTTAP_SMOOTH }, { EIGHTTAP_SMOOTH, EIGHTTAP_REGULAR },
@@ -135,20 +121,6 @@
{ MULTITAP_SHARP, EIGHTTAP_SMOOTH }
};
-static INLINE int mode_offset(const PREDICTION_MODE mode) {
- if (mode >= NEARESTMV) {
- return INTER_OFFSET(mode);
- } else {
- switch (mode) {
- case DC_PRED: return 0;
- case V_PRED: return 1;
- case H_PRED: return 2;
- case SMOOTH_PRED: return 3;
- default: assert(0); return -1;
- }
- }
-}
-
enum {
// INTER_ALL = (1 << NEARESTMV) | (1 << NEARMV) | (1 << NEWMV),
INTER_NEAREST = (1 << NEARESTMV),
@@ -205,19 +177,52 @@
memset(&bp->pmi, 0, sizeof(bp->pmi));
}
-static INLINE int subpel_select(AV1_COMP *cpi, BLOCK_SIZE bsize, int_mv *mv) {
- int mv_thresh = 4;
- const int is_low_resoln =
- (cpi->common.width * cpi->common.height <= 320 * 240);
- mv_thresh = (bsize > BLOCK_32X32) ? 2 : (bsize > BLOCK_16X16) ? 4 : 6;
- if (cpi->rc.avg_frame_low_motion > 0 && cpi->rc.avg_frame_low_motion < 40)
- mv_thresh = 12;
- mv_thresh = (is_low_resoln) ? mv_thresh >> 1 : mv_thresh;
- if (abs(mv->as_fullmv.row) >= mv_thresh ||
- abs(mv->as_fullmv.col) >= mv_thresh)
- return HALF_PEL;
- else
- return cpi->sf.mv_sf.subpel_force_stop;
+static INLINE int subpel_select(AV1_COMP *cpi, MACROBLOCK *x, BLOCK_SIZE bsize,
+ int_mv *mv, MV ref_mv, FULLPEL_MV start_mv,
+ bool fullpel_performed_well) {
+ const int frame_lowmotion = cpi->rc.avg_frame_low_motion;
+ // Reduce MV precision for higher int MV value & frame-level motion
+ if (cpi->sf.rt_sf.reduce_mv_pel_precision_highmotion >= 2) {
+ int mv_thresh = 4;
+ const int is_low_resoln =
+ (cpi->common.width * cpi->common.height <= 320 * 240);
+ mv_thresh = (bsize > BLOCK_32X32) ? 2 : (bsize > BLOCK_16X16) ? 4 : 6;
+ if (frame_lowmotion > 0 && frame_lowmotion < 40) mv_thresh = 12;
+ mv_thresh = (is_low_resoln) ? mv_thresh >> 1 : mv_thresh;
+ if (abs(mv->as_fullmv.row) >= mv_thresh ||
+ abs(mv->as_fullmv.col) >= mv_thresh)
+ return HALF_PEL;
+ } else if (cpi->sf.rt_sf.reduce_mv_pel_precision_highmotion >= 1) {
+ int mv_thresh;
+ if (frame_lowmotion > 0 && frame_lowmotion < 40)
+ mv_thresh = 12;
+ else
+ mv_thresh = (bsize >= BLOCK_32X32) ? 4 : (bsize >= BLOCK_16X16) ? 6 : 8;
+ if (abs(mv->as_fullmv.row) >= (mv_thresh << 1) ||
+ abs(mv->as_fullmv.col) >= (mv_thresh << 1))
+ return FULL_PEL;
+ else if (abs(mv->as_fullmv.row) >= mv_thresh ||
+ abs(mv->as_fullmv.col) >= mv_thresh)
+ return HALF_PEL;
+ }
+ // Reduce MV precision for relatively static (e.g. background), low-complex
+ // large areas
+ if (cpi->sf.rt_sf.reduce_mv_pel_precision_lowcomplex >= 2) {
+ const int qband = x->qindex >> (QINDEX_BITS - 2);
+ assert(qband < 4);
+ if (x->content_state_sb.source_sad_nonrd <= kVeryLowSad &&
+ bsize > BLOCK_16X16 && qband != 0) {
+ if (x->source_variance < 500)
+ return FULL_PEL;
+ else if (x->source_variance < 5000)
+ return HALF_PEL;
+ }
+ } else if (cpi->sf.rt_sf.reduce_mv_pel_precision_lowcomplex >= 1) {
+ if (fullpel_performed_well && ref_mv.row == 0 && ref_mv.col == 0 &&
+ start_mv.row == 0 && start_mv.col == 0)
+ return HALF_PEL;
+ }
+ return cpi->sf.mv_sf.subpel_force_stop;
}
/*!\brief Runs Motion Estimation for a specific block and specific ref frame.
@@ -251,10 +256,11 @@
MACROBLOCKD *xd = &x->e_mbd;
const AV1_COMMON *cm = &cpi->common;
const int num_planes = av1_num_planes(cm);
+ const SPEED_FEATURES *sf = &cpi->sf;
MB_MODE_INFO *mi = xd->mi[0];
struct buf_2d backup_yv12[MAX_MB_PLANE] = { { 0, 0, 0, 0, 0 } };
- int step_param = (cpi->sf.rt_sf.fullpel_search_step_param)
- ? cpi->sf.rt_sf.fullpel_search_step_param
+ int step_param = (sf->rt_sf.fullpel_search_step_param)
+ ? sf->rt_sf.fullpel_search_step_param
: cpi->mv_search_params.mv_step_param;
FULLPEL_MV start_mv;
const int ref = mi->ref_frame[0];
@@ -284,7 +290,7 @@
else
center_mv = tmp_mv->as_mv;
- const SEARCH_METHODS search_method = cpi->sf.mv_sf.search_method;
+ const SEARCH_METHODS search_method = sf->mv_sf.search_method;
const MotionVectorSearchParams *mv_search_params = &cpi->mv_search_params;
const int ref_stride = xd->plane[0].pre[0].stride;
const search_site_config *src_search_sites = av1_get_search_site_config(
@@ -311,27 +317,27 @@
SUBPEL_MOTION_SEARCH_PARAMS ms_params;
av1_make_default_subpel_ms_params(&ms_params, cpi, x, bsize, &ref_mv,
cost_list);
- if (cpi->sf.rt_sf.force_half_pel_block &&
- cpi->sf.mv_sf.subpel_force_stop < HALF_PEL)
- ms_params.forced_stop = subpel_select(cpi, bsize, tmp_mv);
- if (cpi->sf.rt_sf.reduce_zeromv_mvres && ref_mv.row == 0 &&
- ref_mv.col == 0 && start_mv.row == 0 && start_mv.col == 0) {
- // If both the refmv and the fullpel results show zero mv, then there is
- // high likelihood that the current block is static. So we can try to
- // reduce the mv resolution here.
- // These thresholds are the mean var rd collected from multiple encoding
- // runs.
- if ((bsize == BLOCK_64X64 && full_var_rd * 40 < 62267 * 7) ||
- (bsize == BLOCK_32X32 && full_var_rd * 8 < 42380) ||
- (bsize == BLOCK_16X16 && full_var_rd * 8 < 10127)) {
- ms_params.forced_stop = HALF_PEL;
- }
- }
+ const bool fullpel_performed_well =
+ (bsize == BLOCK_64X64 && full_var_rd * 40 < 62267 * 7) ||
+ (bsize == BLOCK_32X32 && full_var_rd * 8 < 42380) ||
+ (bsize == BLOCK_16X16 && full_var_rd * 8 < 10127);
+ if (sf->rt_sf.reduce_mv_pel_precision_highmotion ||
+ sf->rt_sf.reduce_mv_pel_precision_lowcomplex)
+ ms_params.forced_stop = subpel_select(cpi, x, bsize, tmp_mv, ref_mv,
+ start_mv, fullpel_performed_well);
MV subpel_start_mv = get_mv_from_fullmv(&tmp_mv->as_fullmv);
- cpi->mv_search_params.find_fractional_mv_step(
- xd, cm, &ms_params, subpel_start_mv, &tmp_mv->as_mv, &dis,
- &x->pred_sse[ref], NULL);
+ if (sf->rt_sf.use_adaptive_subpel_search &&
+ (fullpel_performed_well ||
+ x->content_state_sb.source_sad_nonrd <= kLowSad)) {
+ av1_find_best_sub_pixel_tree_pruned_more(xd, cm, &ms_params,
+ subpel_start_mv, &tmp_mv->as_mv,
+ &dis, &x->pred_sse[ref], NULL);
+ } else {
+ cpi->mv_search_params.find_fractional_mv_step(
+ xd, cm, &ms_params, subpel_start_mv, &tmp_mv->as_mv, &dis,
+ &x->pred_sse[ref], NULL);
+ }
*rate_mv =
av1_mv_bit_cost(&tmp_mv->as_mv, &ref_mv, x->mv_costs->nmv_joint_cost,
@@ -407,9 +413,12 @@
SUBPEL_MOTION_SEARCH_PARAMS ms_params;
av1_make_default_subpel_ms_params(&ms_params, cpi, x, bsize, &ref_mv, NULL);
- if (cpi->sf.rt_sf.force_half_pel_block &&
- cpi->sf.mv_sf.subpel_force_stop < HALF_PEL)
- ms_params.forced_stop = subpel_select(cpi, bsize, &best_mv);
+ if (cpi->sf.rt_sf.reduce_mv_pel_precision_highmotion ||
+ cpi->sf.rt_sf.reduce_mv_pel_precision_lowcomplex) {
+ FULLPEL_MV start_mv = { .row = 0, .col = 0 };
+ ms_params.forced_stop =
+ subpel_select(cpi, x, bsize, &best_mv, ref_mv, start_mv, false);
+ }
MV start_mv = get_mv_from_fullmv(&best_mv.as_fullmv);
cpi->mv_search_params.find_fractional_mv_step(
xd, cm, &ms_params, start_mv, &best_mv.as_mv, &dis,
@@ -624,7 +633,7 @@
TX_SIZE tx_size;
int k;
- if (x->force_zeromv_skip) {
+ if (x->force_zeromv_skip_for_blk) {
*early_term = 1;
rd_stats->rate = 0;
rd_stats->dist = 0;
@@ -778,7 +787,15 @@
static void model_rd_for_sb_y(const AV1_COMP *const cpi, BLOCK_SIZE bsize,
MACROBLOCK *x, MACROBLOCKD *xd,
- RD_STATS *rd_stats, int calculate_rd) {
+ RD_STATS *rd_stats, unsigned int *var_out,
+ int calculate_rd, int *early_term) {
+ if (x->force_zeromv_skip_for_blk && early_term != NULL) {
+ *early_term = 1;
+ rd_stats->rate = 0;
+ rd_stats->dist = 0;
+ rd_stats->sse = 0;
+ }
+
// Note our transform coeffs are 8 times an orthogonal transform.
// Hence quantizer step is also 8 times. To get effective quantizer
// we need to divide by 8 before sending to modeling function.
@@ -796,6 +813,9 @@
p->src.buf, p->src.stride, pd->dst.buf, pd->dst.stride, &sse);
int force_skip = 0;
xd->mi[0]->tx_size = calculate_tx_size(cpi, bsize, x, var, sse, &force_skip);
+ if (var_out) {
+ *var_out = var;
+ }
if (calculate_rd && (!force_skip || ref == INTRA_FRAME)) {
const int bwide = block_size_wide[bsize];
@@ -913,7 +933,7 @@
* \param[in] tx_type Transform kernel type
* \param[in] is_inter_mode Flag to indicate inter mode
*
- * \return Nothing is returned. Instead, calculated RD cost is placed to
+ * \remark Nothing is returned. Instead, calculated RD cost is placed to
* \c this_rdc. \c skippable flag is set if there is no non-zero quantized
* coefficients for Hadamard transform
*/
@@ -1421,7 +1441,7 @@
* \param[in] arg Pointer to a structure that holds paramaters
* for intra mode search
*
- * \return Nothing is returned. Instead, best mode and RD Cost of the best mode
+ * \remark Nothing is returned. Instead, best mode and RD Cost of the best mode
* are set in \c args->rdc and \c args->mode
*/
static void estimate_block_intra(int plane, int block, int row, int col,
@@ -1531,7 +1551,8 @@
mi->interp_filters = av1_broadcast_interp_filter(EIGHTTAP_REGULAR);
xd->plane[0].pre[0] = yv12_mb[LAST_FRAME][0];
av1_enc_build_inter_predictor_y(xd, mi_row, mi_col);
- model_rd_for_sb_y(cpi, bsize, x, xd, &this_rdc, 1);
+ unsigned int var;
+ model_rd_for_sb_y(cpi, bsize, x, xd, &this_rdc, &var, 1, NULL);
const int16_t mode_ctx =
av1_mode_context_analyzer(mbmi_ext->mode_context, mi->ref_frame);
@@ -1597,11 +1618,13 @@
* for prediction re-use
* \param[out] this_early_term Flag, indicating that transform can be
* skipped
+ * \param[out] var The residue variance of the current
+ * predictor.
* \param[in] use_model_yrd_large Flag, indicating special logic to handle
* large blocks
* \param[in] best_sse Best sse so far.
*
- * \return Nothing is returned. Instead, calculated RD cost is placed to
+ * \remark Nothing is returned. Instead, calculated RD cost is placed to
* \c this_rdc and best filter is placed to \c mi->interp_filters. In case
* \c reuse_inter_pred flag is set, this function also ouputs
* \c this_mode_pred. Also \c this_early_temp is set if transform can be
@@ -1611,8 +1634,8 @@
int mi_row, int mi_col, PRED_BUFFER *tmp,
BLOCK_SIZE bsize, int reuse_inter_pred,
PRED_BUFFER **this_mode_pred,
- int *this_early_term, int use_model_yrd_large,
- int64_t best_sse) {
+ int *this_early_term, unsigned int *var,
+ int use_model_yrd_large, int64_t best_sse) {
AV1_COMMON *const cm = &cpi->common;
MACROBLOCKD *const xd = &x->e_mbd;
struct macroblockd_plane *const pd = &xd->plane[0];
@@ -1635,17 +1658,19 @@
mi->interp_filters.as_filters.x_filter = filters_ref_set[i].filter_x;
mi->interp_filters.as_filters.y_filter = filters_ref_set[i].filter_y;
av1_enc_build_inter_predictor_y(xd, mi_row, mi_col);
+ unsigned int curr_var = UINT_MAX;
if (use_model_yrd_large)
model_skip_for_sb_y_large(cpi, bsize, mi_row, mi_col, x, xd,
&pf_rd_stats[i], this_early_term, 1, best_sse,
- NULL, UINT_MAX);
+ &curr_var, UINT_MAX);
else
- model_rd_for_sb_y(cpi, bsize, x, xd, &pf_rd_stats[i], 1);
+ model_rd_for_sb_y(cpi, bsize, x, xd, &pf_rd_stats[i], &curr_var, 1, NULL);
pf_rd_stats[i].rate += av1_get_switchable_rate(
x, xd, cm->features.interp_filter, cm->seq_params->enable_dual_filter);
cost = RDCOST(x->rdmult, pf_rd_stats[i].rate, pf_rd_stats[i].dist);
pf_tx_size[i] = mi->tx_size;
if (cost < best_cost) {
+ *var = curr_var;
best_filter_index = i;
best_cost = cost;
best_skip = pf_rd_stats[i].skip_txfm;
@@ -1786,7 +1811,7 @@
&pf_rd_stats[i], this_early_term, 1, best_sse,
NULL, UINT_MAX);
else
- model_rd_for_sb_y(cpi, bsize, x, xd, &pf_rd_stats[i], 1);
+ model_rd_for_sb_y(cpi, bsize, x, xd, &pf_rd_stats[i], NULL, 1, NULL);
pf_rd_stats[i].rate +=
av1_get_switchable_rate(x, xd, cm->features.interp_filter,
cm->seq_params->enable_dual_filter);
@@ -1849,7 +1874,7 @@
&pf_rd_stats[i], this_early_term, 1,
best_sse, NULL, UINT_MAX);
else
- model_rd_for_sb_y(cpi, bsize, x, xd, &pf_rd_stats[i], 1);
+ model_rd_for_sb_y(cpi, bsize, x, xd, &pf_rd_stats[i], NULL, 1, NULL);
pf_rd_stats[i].rate +=
mode_costs->motion_mode_cost[bsize][mi->motion_mode];
@@ -1881,11 +1906,13 @@
#endif // !CONFIG_REALTIME_ONLY
#define COLLECT_PICK_MODE_STAT 0
+#define COLLECT_NON_SQR_STAT 0
#if COLLECT_PICK_MODE_STAT
+#include "aom_ports/aom_timer.h"
typedef struct _mode_search_stat {
int32_t num_blocks[BLOCK_SIZES];
- int64_t avg_block_times[BLOCK_SIZES];
+ int64_t total_block_times[BLOCK_SIZES];
int32_t num_searches[BLOCK_SIZES][MB_MODE_COUNT];
int32_t num_nonskipped_searches[BLOCK_SIZES][MB_MODE_COUNT];
int64_t search_times[BLOCK_SIZES][MB_MODE_COUNT];
@@ -1896,8 +1923,15 @@
int64_t txfm_time[BLOCK_SIZES][MB_MODE_COUNT];
struct aom_usec_timer timer1;
struct aom_usec_timer timer2;
- struct aom_usec_timer timer3;
+ struct aom_usec_timer bsize_timer;
} mode_search_stat;
+
+static void AOM_INLINE print_stage_time(const char *stage_name,
+ int64_t stage_time,
+ int64_t total_time) {
+ printf(" %s: %ld (%f%%)\n", stage_name, stage_time,
+ 100 * stage_time / (float)total_time);
+}
#endif // COLLECT_PICK_MODE_STAT
static void compute_intra_yprediction(const AV1_COMMON *cm,
@@ -1972,6 +2006,17 @@
// mode tests.
for (int i = 0; i < 4; ++i) {
PREDICTION_MODE this_mode = intra_mode_list[i];
+
+ // As per the statistics generated for intra mode evaluation in the nonrd
+ // path, it is found that the probability of H_PRED mode being the winner is
+ // very less when the best mode so far is V_PRED (out of DC_PRED and
+ // V_PRED). If V_PRED is the winner mode out of DC_PRED and V_PRED, it could
+ // imply the presence of a vertically dominant pattern. Hence, H_PRED mode
+ // is not evaluated.
+ if (cpi->sf.rt_sf.prune_h_pred_using_best_mode_so_far &&
+ this_mode == H_PRED && best_mode == V_PRED)
+ continue;
+
this_rdc.dist = this_rdc.rate = 0;
args.mode = this_mode;
args.skippable = 1;
@@ -2061,7 +2106,7 @@
}
if (use_last_ref_frame &&
- (x->nonrd_prune_ref_frame_search > 2 || x->force_zeromv_skip ||
+ (x->nonrd_prune_ref_frame_search > 2 || x->force_zeromv_skip_for_blk ||
(x->nonrd_prune_ref_frame_search > 1 && bsize > BLOCK_64X64))) {
use_golden_ref_frame = 0;
use_alt_ref_frame = 0;
@@ -2082,8 +2127,15 @@
}
// Skip golden reference if color is set, on flat blocks with motion.
- if (x->source_variance < 500 &&
- x->content_state_sb.source_sad_nonrd > kLowSad &&
+ // For screen: always skip golden (if color_sensitivity_sb_g is set)
+ // except when x->nonrd_prune_ref_frame_search = 0. This latter flag
+ // may be set in the variance partition when golden is a much beter
+ // reference than last, in which case it may not be worth skipping
+ // golden completely.
+ if (((cpi->oxcf.tune_cfg.content == AOM_CONTENT_SCREEN &&
+ x->nonrd_prune_ref_frame_search != 0) ||
+ (x->source_variance < 500 &&
+ x->content_state_sb.source_sad_nonrd > kLowSad)) &&
(x->color_sensitivity_sb_g[0] == 1 || x->color_sensitivity_sb_g[1] == 1))
use_golden_ref_frame = 0;
@@ -2116,8 +2168,6 @@
* \param[in] x Pointer to structure holding all the
* data for the current macroblock
* \param[in] bsize Current block size
- * \param[in] use_modeled_non_rd_cost Flag, indicating usage of curvfit
- * model for RD cost
* \param[in] best_early_term Flag, indicating that TX for the
* best inter mode was skipped
* \param[in] ref_cost_intra Cost of signalling intra mode
@@ -2134,14 +2184,13 @@
* \param[in] ctx Pointer to structure holding coding
* contexts and modes for the block
*
- * \return Nothing is returned. Instead, calculated RD cost is placed to
+ * \remark Nothing is returned. Instead, calculated RD cost is placed to
* \c best_rdc and best selected mode is placed to \c best_pickmode
*/
static void estimate_intra_mode(
- AV1_COMP *cpi, MACROBLOCK *x, BLOCK_SIZE bsize, int use_modeled_non_rd_cost,
- int best_early_term, unsigned int ref_cost_intra, int reuse_prediction,
- struct buf_2d *orig_dst, PRED_BUFFER *tmp_buffers,
- PRED_BUFFER **this_mode_pred, RD_STATS *best_rdc,
+ AV1_COMP *cpi, MACROBLOCK *x, BLOCK_SIZE bsize, int best_early_term,
+ unsigned int ref_cost_intra, int reuse_prediction, struct buf_2d *orig_dst,
+ PRED_BUFFER *tmp_buffers, PRED_BUFFER **this_mode_pred, RD_STATS *best_rdc,
BEST_PICKMODE *best_pickmode, PICK_MODE_CONTEXT *ctx) {
AV1_COMMON *const cm = &cpi->common;
MACROBLOCKD *const xd = &x->e_mbd;
@@ -2150,6 +2199,8 @@
const unsigned char segment_id = mi->segment_id;
const int *const rd_threshes = cpi->rd.threshes[segment_id][bsize];
const int *const rd_thresh_freq_fact = x->thresh_freq_fact[bsize];
+ const bool is_screen_content =
+ cpi->oxcf.tune_cfg.content == AOM_CONTENT_SCREEN;
const int mi_row = xd->mi_row;
const int mi_col = xd->mi_col;
struct macroblockd_plane *const pd = &xd->plane[0];
@@ -2161,7 +2212,8 @@
int intra_cost_penalty = av1_get_intra_cost_penalty(
quant_params->base_qindex, quant_params->y_dc_delta_q,
cm->seq_params->bit_depth);
- int64_t inter_mode_thresh = RDCOST(x->rdmult, intra_cost_penalty, 0);
+ int64_t inter_mode_thresh =
+ RDCOST(x->rdmult, ref_cost_intra + intra_cost_penalty, 0);
int perform_intra_pred = cpi->sf.rt_sf.check_intra_pred_nonrd;
int force_intra_check = 0;
// For spatial enhancemanent layer: turn off intra prediction if the
@@ -2195,13 +2247,13 @@
abs(mi->mv[0].as_mv.row) >= motion_thresh ||
abs(mi->mv[0].as_mv.col) >= motion_thresh)) {
intra_cost_penalty = intra_cost_penalty >> 2;
- inter_mode_thresh = RDCOST(x->rdmult, intra_cost_penalty, 0);
+ inter_mode_thresh =
+ RDCOST(x->rdmult, ref_cost_intra + intra_cost_penalty, 0);
do_early_exit_rdthresh = 0;
}
if ((x->source_variance < AOMMAX(50, (spatial_var_thresh >> 1)) &&
x->content_state_sb.source_sad_nonrd >= kHighSad) ||
- (cpi->oxcf.tune_cfg.content == AOM_CONTENT_SCREEN &&
- x->source_variance == 0 &&
+ (is_screen_content && x->source_variance < 50 &&
((bsize >= BLOCK_32X32 &&
x->content_state_sb.source_sad_nonrd != kZeroSad) ||
x->color_sensitivity[0] == 1 || x->color_sensitivity[1] == 1)))
@@ -2210,7 +2262,7 @@
// even if best_early_term is set.
if (bsize >= BLOCK_32X32) best_early_term = 0;
} else if (cpi->sf.rt_sf.source_metrics_sb_nonrd &&
- x->content_state_sb.source_sad_nonrd == kLowSad) {
+ x->content_state_sb.source_sad_nonrd <= kLowSad) {
perform_intra_pred = 0;
}
@@ -2223,19 +2275,25 @@
if (!(best_rdc->rdcost == INT64_MAX || force_intra_check ||
(perform_intra_pred && !best_early_term &&
- best_rdc->rdcost > inter_mode_thresh &&
bsize <= cpi->sf.part_sf.max_intra_bsize))) {
return;
}
+ // Early exit based on RD cost calculated using known rate. When
+ // is_screen_content is true, more bias is given to intra modes. Hence,
+ // considered conservative threshold in early exit for the same.
+ const int64_t known_rd = is_screen_content
+ ? CALC_BIASED_RDCOST(inter_mode_thresh)
+ : inter_mode_thresh;
+ if (known_rd > best_rdc->rdcost) return;
+
struct estimate_block_intra_args args = { cpi, x, DC_PRED, 1, 0 };
TX_SIZE intra_tx_size = AOMMIN(
AOMMIN(max_txsize_lookup[bsize],
tx_mode_to_biggest_tx_size[txfm_params->tx_mode_search_type]),
TX_16X16);
- if (cpi->oxcf.tune_cfg.content == AOM_CONTENT_SCREEN &&
- cpi->rc.high_source_sad && x->source_variance > spatial_var_thresh &&
- bsize <= BLOCK_16X16)
+ if (is_screen_content && cpi->rc.high_source_sad &&
+ x->source_variance > spatial_var_thresh && bsize <= BLOCK_16X16)
intra_tx_size = TX_4X4;
PRED_BUFFER *const best_pred = best_pickmode->best_pred;
@@ -2257,22 +2315,20 @@
const THR_MODES mode_index = mode_idx[INTRA_FRAME][mode_offset(this_mode)];
const int64_t mode_rd_thresh = rd_threshes[mode_index];
- if (i > 2 || !(force_intra_check == 1 &&
- best_pickmode->best_ref_frame != INTRA_FRAME)) {
+ if (i > 2 || force_intra_check == 0) {
if (!((1 << this_mode) &
cpi->sf.rt_sf.intra_y_mode_bsize_mask_nrd[bsize]))
continue;
}
- if (cpi->oxcf.tune_cfg.content == AOM_CONTENT_SCREEN &&
- cpi->sf.rt_sf.source_metrics_sb_nonrd) {
+ if (is_screen_content && cpi->sf.rt_sf.source_metrics_sb_nonrd) {
// For spatially flat blocks with zero motion only check
// DC mode.
if (x->content_state_sb.source_sad_nonrd == kZeroSad &&
x->source_variance == 0 && this_mode != DC_PRED)
continue;
- // Only test Intra for big blocks if spatial_variance is 0.
- else if (bsize > BLOCK_32X32 && x->source_variance > 0)
+ // Only test Intra for big blocks if spatial_variance is small.
+ else if (bsize > BLOCK_32X32 && x->source_variance > 50)
continue;
}
@@ -2295,11 +2351,8 @@
mi->tx_size = intra_tx_size;
compute_intra_yprediction(cm, this_mode, bsize, x, xd);
// Look into selecting tx_size here, based on prediction residual.
- if (use_modeled_non_rd_cost)
- model_rd_for_sb_y(cpi, bsize, x, xd, &this_rdc, 1);
- else
- av1_block_yrd(cpi, x, mi_row, mi_col, &this_rdc, &args.skippable, bsize,
- mi->tx_size, DCT_DCT, 0);
+ av1_block_yrd(cpi, x, mi_row, mi_col, &this_rdc, &args.skippable, bsize,
+ mi->tx_size, DCT_DCT, 0);
// TODO(kyslov@) Need to account for skippable
if (x->color_sensitivity[0]) {
av1_foreach_transformed_block_in_plane(xd, uv_bsize, 1,
@@ -2325,13 +2378,12 @@
this_rdc.rate += mode_cost;
this_rdc.rdcost = RDCOST(x->rdmult, this_rdc.rate, this_rdc.dist);
- if (cpi->oxcf.tune_cfg.content == AOM_CONTENT_SCREEN &&
- cpi->sf.rt_sf.source_metrics_sb_nonrd) {
+ if (is_screen_content && cpi->sf.rt_sf.source_metrics_sb_nonrd) {
// For blocks with low spatial variance and color sad,
// favor the intra-modes, only on scene/slide change.
if (cpi->rc.high_source_sad && x->source_variance < 800 &&
(x->color_sensitivity[0] || x->color_sensitivity[1]))
- this_rdc.rdcost = (7 * this_rdc.rdcost) >> 3;
+ this_rdc.rdcost = CALC_BIASED_RDCOST(this_rdc.rdcost);
// Otherwise bias against intra for blocks with zero
// motion and no color, on non-scene/slide changes.
else if (!cpi->rc.high_source_sad && x->source_variance > 0 &&
@@ -2550,64 +2602,28 @@
}
}
-static int skip_comp_based_on_sad(AV1_COMP *cpi, MACROBLOCK *x,
- const int mi_row, const int mi_col,
- BLOCK_SIZE bsize) {
- AV1_COMMON *const cm = &cpi->common;
- assert(!(mi_row % 16) && !(mi_col % 16));
- const int sb_size_by_mb = (cm->seq_params->sb_size == BLOCK_128X128)
- ? (cm->seq_params->mib_size >> 1)
- : cm->seq_params->mib_size;
- const int sb_cols =
- (cm->mi_params.mi_cols + sb_size_by_mb - 1) / sb_size_by_mb;
- const uint64_t sad_skp_comp_th[2][3] = { { 2700, 3100 }, // CPU 9
- { 2700, 3200 } }; // CPU 10
- const uint64_t sad_blkwise_var_th = 5000;
- const float qindex_th_scale[5] = { 0.75f, 0.9f, 1.0f, 1.1f, 1.25f };
- const int qindex_band = (5 * x->qindex) >> QINDEX_BITS;
- assert(qindex_band < 5);
- const int sp_idx = (cpi->sf.rt_sf.sad_based_comp_prune >= 2);
- const int bsize_idx = (bsize == BLOCK_128X128);
- const uint64_t sad_skp_comp_th_val = (uint64_t)(
- sad_skp_comp_th[sp_idx][bsize_idx] * qindex_th_scale[qindex_band]);
- uint64_t blk_sad = 0, sad00, sad01, sad10, sad11, min_sad, max_sad;
- const int sbi_col = mi_col / 16;
- const int sbi_row = mi_row / 16;
- const uint64_t *cur_blk_sad =
- &cpi->src_sad_blk_64x64[sbi_col + sbi_row * sb_cols];
-
- if (bsize == BLOCK_128X128) {
- sad00 = cur_blk_sad[0];
- sad01 = cur_blk_sad[1];
- sad10 = cur_blk_sad[sb_cols];
- sad11 = cur_blk_sad[1 + sb_cols];
- min_sad = AOMMIN(AOMMIN(AOMMIN(sad00, sad01), sad10), sad11);
- max_sad = AOMMAX(AOMMAX(AOMMAX(sad00, sad01), sad10), sad11);
- if (max_sad - min_sad > sad_blkwise_var_th) return 0;
- blk_sad = (sad00 + sad01 + sad10 + sad11 + 2) >> 2;
- } else if (bsize == BLOCK_128X64) {
- sad00 = cur_blk_sad[0];
- sad01 = cur_blk_sad[1];
- min_sad = AOMMIN(sad00, sad01);
- max_sad = AOMMAX(sad00, sad01);
- if (max_sad - min_sad > sad_blkwise_var_th) return 0;
- blk_sad = (sad00 + sad01 + 1) >> 1;
- } else if (bsize == BLOCK_64X128) {
- sad00 = cur_blk_sad[0];
- sad10 = cur_blk_sad[sb_cols];
- min_sad = AOMMIN(sad00, sad10);
- max_sad = AOMMAX(sad00, sad10);
- if (max_sad - min_sad > sad_blkwise_var_th) return 0;
- blk_sad = (sad00 + sad10 + 1) >> 1;
- } else if (bsize <= BLOCK_64X64) {
- blk_sad = cur_blk_sad[0];
- } else {
- assert(0);
+// Prune compound mode if the single mode variance is lower than a fixed
+// percentage of the median value.
+static bool skip_comp_based_on_var(
+ const unsigned int (*single_vars)[REF_FRAMES], BLOCK_SIZE bsize) {
+ unsigned int best_var = UINT_MAX;
+ for (int cur_mode_idx = 0; cur_mode_idx < RTC_INTER_MODES; cur_mode_idx++) {
+ for (int ref_idx = 0; ref_idx < REF_FRAMES; ref_idx++) {
+ best_var = AOMMIN(best_var, single_vars[cur_mode_idx][ref_idx]);
+ }
}
+ const unsigned int thresh_64 = (unsigned int)(0.57356805f * 8659);
+ const unsigned int thresh_32 = (unsigned int)(0.23964763f * 4281);
- if (blk_sad < sad_skp_comp_th_val) return 1;
-
- return 0;
+ // Currently, the thresh for 128 and 16 are not well-tuned. We are using the
+ // results from 64 and 32 as an heuristic.
+ switch (bsize) {
+ case BLOCK_128X128: return best_var < 4 * thresh_64;
+ case BLOCK_64X64: return best_var < thresh_64;
+ case BLOCK_32X32: return best_var < thresh_32;
+ case BLOCK_16X16: return best_var < thresh_32 / 4;
+ default: return false;
+ }
}
static AOM_FORCE_INLINE void fill_single_inter_mode_costs(
@@ -2700,6 +2716,62 @@
return 1;
}
+static AOM_INLINE bool previous_mode_performed_poorly(
+ PREDICTION_MODE mode, MV_REFERENCE_FRAME ref_frame,
+ const unsigned int (*vars)[REF_FRAMES],
+ const int64_t (*uv_dist)[REF_FRAMES]) {
+ unsigned int best_var = UINT_MAX;
+ int64_t best_uv_dist = INT64_MAX;
+ for (int midx = 0; midx < RTC_INTER_MODES; midx++) {
+ best_var = AOMMIN(best_var, vars[midx][ref_frame]);
+ best_uv_dist = AOMMIN(best_uv_dist, uv_dist[midx][ref_frame]);
+ }
+ assert(best_var != UINT_MAX && "Invalid variance data.");
+ const float mult = 1.125f;
+ bool var_bad = mult * best_var < vars[INTER_OFFSET(mode)][ref_frame];
+ if (uv_dist[INTER_OFFSET(mode)][ref_frame] < INT64_MAX &&
+ best_uv_dist != uv_dist[INTER_OFFSET(mode)][ref_frame]) {
+ // If we have chroma info, then take it into account
+ var_bad &= mult * best_uv_dist < uv_dist[INTER_OFFSET(mode)][ref_frame];
+ }
+ return var_bad;
+}
+
+static AOM_INLINE bool prune_compoundmode_with_singlemode_var(
+ PREDICTION_MODE compound_mode, MV_REFERENCE_FRAME ref_frame,
+ MV_REFERENCE_FRAME ref_frame2, const int_mv (*frame_mv)[REF_FRAMES],
+ const uint8_t (*mode_checked)[REF_FRAMES],
+ const unsigned int (*vars)[REF_FRAMES],
+ const int64_t (*uv_dist)[REF_FRAMES]) {
+ const PREDICTION_MODE single_mode0 = compound_ref0_mode(compound_mode);
+ const PREDICTION_MODE single_mode1 = compound_ref1_mode(compound_mode);
+
+ bool first_ref_valid = false, second_ref_valid = false;
+ bool first_ref_bad = false, second_ref_bad = false;
+ if (mode_checked[single_mode0][ref_frame] &&
+ frame_mv[single_mode0][ref_frame].as_int ==
+ frame_mv[compound_mode][ref_frame].as_int &&
+ vars[INTER_OFFSET(single_mode0)][ref_frame] < UINT_MAX) {
+ first_ref_valid = true;
+ first_ref_bad =
+ previous_mode_performed_poorly(single_mode0, ref_frame, vars, uv_dist);
+ }
+ if (mode_checked[single_mode1][ref_frame2] &&
+ frame_mv[single_mode1][ref_frame2].as_int ==
+ frame_mv[compound_mode][ref_frame2].as_int &&
+ vars[INTER_OFFSET(single_mode1)][ref_frame2] < UINT_MAX) {
+ second_ref_valid = true;
+ second_ref_bad =
+ previous_mode_performed_poorly(single_mode1, ref_frame2, vars, uv_dist);
+ }
+ if (first_ref_valid && second_ref_valid) {
+ return first_ref_bad && second_ref_bad;
+ } else if (first_ref_valid || second_ref_valid) {
+ return first_ref_bad || second_ref_bad;
+ }
+ return false;
+}
+
void av1_nonrd_pick_inter_mode_sb(AV1_COMP *cpi, TileDataEnc *tile_data,
MACROBLOCK *x, RD_STATS *rd_cost,
BLOCK_SIZE bsize, PICK_MODE_CONTEXT *ctx) {
@@ -2734,7 +2806,7 @@
int use_zeromv =
cpi->oxcf.tune_cfg.content == AOM_CONTENT_SCREEN ||
((cpi->oxcf.speed >= 9 && cpi->rc.avg_frame_low_motion > 70) ||
- cpi->sf.rt_sf.nonrd_agressive_skip || x->force_zeromv_skip);
+ cpi->sf.rt_sf.nonrd_agressive_skip || x->force_zeromv_skip_for_blk);
int skip_pred_mv = 0;
const int num_inter_modes =
use_zeromv ? NUM_INTER_MODES_REDUCED : NUM_INTER_MODES_RT;
@@ -2755,7 +2827,7 @@
const TxfmSearchParams *txfm_params = &x->txfm_search_params;
TxfmSearchInfo *txfm_info = &x->txfm_search_info;
#if COLLECT_PICK_MODE_STAT
- aom_usec_timer_start(&ms_stat.timer2);
+ aom_usec_timer_start(&ms_stat.bsize_timer);
#endif
int64_t thresh_sad_pred = INT64_MAX;
const int mi_row = xd->mi_row;
@@ -2763,12 +2835,15 @@
int svc_mv_col = 0;
int svc_mv_row = 0;
int force_mv_inter_layer = 0;
- int use_modeled_non_rd_cost = 0;
bool comp_use_zero_zeromv_only = 0;
int tot_num_comp_modes = NUM_COMP_INTER_MODES_RT;
- unsigned int zeromv_var[REF_FRAMES];
- for (int idx = 0; idx < REF_FRAMES; idx++) {
- zeromv_var[idx] = UINT_MAX;
+ unsigned int vars[RTC_INTER_MODES][REF_FRAMES];
+ int64_t uv_dist[RTC_INTER_MODES][REF_FRAMES];
+ for (int idx = 0; idx < RTC_INTER_MODES; idx++) {
+ for (int ref = 0; ref < REF_FRAMES; ref++) {
+ vars[idx][ref] = UINT_MAX;
+ uv_dist[idx][ref] = INT64_MAX;
+ }
}
#if CONFIG_AV1_TEMPORAL_DENOISING
const int denoise_recheck_zeromv = 1;
@@ -2838,7 +2913,7 @@
get_ref_frame_use_mask(cpi, x, mi, mi_row, mi_col, bsize, gf_temporal_ref,
use_ref_frame_mask, &force_skip_low_temp_var);
- skip_pred_mv = x->force_zeromv_skip ||
+ skip_pred_mv = x->force_zeromv_skip_for_blk ||
(x->nonrd_prune_ref_frame_search > 2 &&
x->color_sensitivity[0] != 2 && x->color_sensitivity[1] != 2);
@@ -2854,13 +2929,6 @@
tot_num_comp_modes = 0;
}
- // Skip compound mode based on sad
- if (tot_num_comp_modes && cpi->sf.rt_sf.sad_based_comp_prune &&
- bsize >= BLOCK_64X64 && cpi->src_sad_blk_64x64 &&
- skip_comp_based_on_sad(cpi, x, mi_row, mi_col, bsize)) {
- tot_num_comp_modes = 0;
- }
-
for (MV_REFERENCE_FRAME ref_frame_iter = LAST_FRAME;
ref_frame_iter <= ALTREF_FRAME; ++ref_frame_iter) {
if (use_ref_frame_mask[ref_frame_iter]) {
@@ -2883,21 +2951,6 @@
const int enable_filter_search =
is_filter_search_enabled(cpi, mi_row, mi_col, bsize, segment_id);
- // TODO(marpan): Look into reducing these conditions. For now constrain
- // it to avoid significant bdrate loss.
- if (cpi->sf.rt_sf.use_modeled_non_rd_cost) {
- if (cpi->svc.non_reference_frame)
- use_modeled_non_rd_cost = 1;
- else if (cpi->svc.number_temporal_layers > 1 &&
- cpi->svc.temporal_layer_id == 0)
- use_modeled_non_rd_cost = 0;
- else
- use_modeled_non_rd_cost =
- (quant_params->base_qindex > 120 && x->source_variance > 100 &&
- bsize <= BLOCK_16X16 && !x->content_state_sb.lighting_change &&
- x->content_state_sb.source_sad_nonrd != kHighSad);
- }
-
#if COLLECT_PICK_MODE_STAT
ms_stat.num_blocks[bsize]++;
#endif
@@ -2917,6 +2970,13 @@
MV_REFERENCE_FRAME last_comp_ref_frame = NONE_FRAME;
for (int idx = 0; idx < num_inter_modes + tot_num_comp_modes; ++idx) {
+ // If we are at the first compound mode, and the single modes already
+ // perform well, then end the search.
+ if (cpi->sf.rt_sf.skip_compound_based_on_var && idx == num_inter_modes &&
+ skip_comp_based_on_var(vars, bsize)) {
+ break;
+ }
+
const struct segmentation *const seg = &cm->seg;
int rate_mv = 0;
@@ -2960,10 +3020,20 @@
if (!use_ref_frame_mask[ref_frame]) continue;
- if (x->force_zeromv_skip &&
- (this_mode != GLOBALMV || ref_frame != LAST_FRAME))
+ if (x->force_zeromv_skip_for_blk &&
+ ((!(this_mode == NEARESTMV &&
+ frame_mv[this_mode][ref_frame].as_int == 0) &&
+ this_mode != GLOBALMV) ||
+ ref_frame != LAST_FRAME))
continue;
+ if (cpi->sf.rt_sf.prune_compoundmode_with_singlemode_var && comp_pred &&
+ prune_compoundmode_with_singlemode_var(this_mode, ref_frame, ref_frame2,
+ frame_mv, mode_checked, vars,
+ uv_dist)) {
+ continue;
+ }
+
force_mv_inter_layer = 0;
if (cpi->ppi->use_svc && svc->spatial_layer_id > 0 &&
((ref_frame == LAST_FRAME && svc->skip_mvsearch_last) ||
@@ -3116,24 +3186,29 @@
pd->dst.stride = bw;
}
}
-#if COLLECT_PICK_MODE_STAT
- ms_stat.num_nonskipped_searches[bsize][this_mode]++;
-#endif
if (idx == 0 && !skip_pred_mv) {
// Set color sensitivity on first tested mode only.
// Use y-sad already computed in find_predictors: take the sad with motion
// vector closest to 0; the uv-sad computed below in set_color_sensitivity
// is for zeromv.
- int y_sad = x->pred_mv0_sad[LAST_FRAME];
- if (x->pred_mv1_sad[LAST_FRAME] != INT_MAX &&
- (abs(frame_mv[NEARMV][LAST_FRAME].as_mv.col) +
- abs(frame_mv[NEARMV][LAST_FRAME].as_mv.row)) <
- (abs(frame_mv[NEARESTMV][LAST_FRAME].as_mv.col) +
- abs(frame_mv[NEARESTMV][LAST_FRAME].as_mv.row)))
- y_sad = x->pred_mv1_sad[LAST_FRAME];
- set_color_sensitivity(cpi, x, bsize, y_sad, x->source_variance,
- yv12_mb[LAST_FRAME]);
+ // For screen: first check if golden reference is being used, if so,
+ // force color_sensitivity on if the color sensitivity for sb_g is on.
+ if (cpi->oxcf.tune_cfg.content == AOM_CONTENT_SCREEN &&
+ use_ref_frame_mask[GOLDEN_FRAME]) {
+ if (x->color_sensitivity_sb_g[0] == 1) x->color_sensitivity[0] = 1;
+ if (x->color_sensitivity_sb_g[1] == 1) x->color_sensitivity[1] = 1;
+ } else {
+ int y_sad = x->pred_mv0_sad[LAST_FRAME];
+ if (x->pred_mv1_sad[LAST_FRAME] != INT_MAX &&
+ (abs(frame_mv[NEARMV][LAST_FRAME].as_mv.col) +
+ abs(frame_mv[NEARMV][LAST_FRAME].as_mv.row)) <
+ (abs(frame_mv[NEARESTMV][LAST_FRAME].as_mv.col) +
+ abs(frame_mv[NEARESTMV][LAST_FRAME].as_mv.row)))
+ y_sad = x->pred_mv1_sad[LAST_FRAME];
+ set_color_sensitivity(cpi, x, bsize, y_sad, x->source_variance,
+ yv12_mb[LAST_FRAME]);
+ }
}
mi->motion_mode = SIMPLE_TRANSLATION;
#if !CONFIG_REALTIME_ONLY
@@ -3150,6 +3225,7 @@
#endif
search_filter_ref(cpi, x, &this_rdc, mi_row, mi_col, tmp, bsize,
reuse_inter_pred, &this_mode_pred, &this_early_term,
+ &vars[INTER_OFFSET(this_mode)][ref_frame],
use_model_yrd_large, best_pickmode.best_sse);
#if COLLECT_PICK_MODE_STAT
aom_usec_timer_mark(&ms_stat.timer2);
@@ -3196,28 +3272,33 @@
av1_enc_build_inter_predictor(cm, xd, mi_row, mi_col, NULL, bsize, 0,
0);
+ unsigned int var_threshold = UINT_MAX;
+ if (cpi->sf.rt_sf.prune_compoundmode_with_singlecompound_var &&
+ comp_pred && use_model_yrd_large) {
+ const PREDICTION_MODE single_mode0 = compound_ref0_mode(this_mode);
+ const PREDICTION_MODE single_mode1 = compound_ref1_mode(this_mode);
+ var_threshold =
+ AOMMIN(var_threshold, vars[INTER_OFFSET(single_mode0)][ref_frame]);
+ var_threshold =
+ AOMMIN(var_threshold, vars[INTER_OFFSET(single_mode1)][ref_frame2]);
+ }
if (use_model_yrd_large) {
- unsigned int var_threshold = UINT_MAX;
- if (cpi->sf.rt_sf.prune_global_globalmv_with_zeromv &&
- this_mode == GLOBAL_GLOBALMV) {
- var_threshold = AOMMIN(var_threshold, zeromv_var[ref_frame]);
- var_threshold = AOMMIN(var_threshold, zeromv_var[ref_frame2]);
- }
-
model_skip_for_sb_y_large(cpi, bsize, mi_row, mi_col, x, xd, &this_rdc,
- &this_early_term, use_modeled_non_rd_cost,
- best_pickmode.best_sse, &var, var_threshold);
- if (!comp_pred && frame_mv[this_mode][ref_frame].as_int == 0) {
- zeromv_var[ref_frame] = var;
- } else if (this_mode == GLOBAL_GLOBALMV) {
- if (var > var_threshold) {
- if (reuse_inter_pred) free_pred_buffer(this_mode_pred);
- continue;
- }
- }
+ &this_early_term, 0, best_pickmode.best_sse,
+ &var, var_threshold);
} else {
- model_rd_for_sb_y(cpi, bsize, x, xd, &this_rdc,
- use_modeled_non_rd_cost);
+ model_rd_for_sb_y(cpi, bsize, x, xd, &this_rdc, &var, 0,
+ &this_early_term);
+ }
+ if (!comp_pred) {
+ vars[INTER_OFFSET(this_mode)][ref_frame] = var;
+ if (frame_mv[this_mode][ref_frame].as_int == 0) {
+ vars[INTER_OFFSET(GLOBALMV)][ref_frame] = var;
+ }
+ }
+ if (comp_pred && var > var_threshold) {
+ if (reuse_inter_pred) free_pred_buffer(this_mode_pred);
+ continue;
}
#if COLLECT_PICK_MODE_STAT
aom_usec_timer_mark(&ms_stat.timer2);
@@ -3240,6 +3321,10 @@
continue;
}
+#if COLLECT_PICK_MODE_STAT
+ ms_stat.num_nonskipped_searches[bsize][this_mode]++;
+#endif
+
const int skip_ctx = av1_get_skip_txfm_context(xd);
const int skip_txfm_cost = mode_costs->skip_txfm_cost[skip_ctx][1];
const int no_skip_txfm_cost = mode_costs->skip_txfm_cost[skip_ctx][0];
@@ -3249,33 +3334,25 @@
this_rdc.rate = skip_txfm_cost;
this_rdc.dist = this_rdc.sse << 4;
} else {
- if (use_modeled_non_rd_cost) {
- if (this_rdc.skip_txfm) {
- this_rdc.rate = skip_txfm_cost;
- } else {
- this_rdc.rate += no_skip_txfm_cost;
- }
- } else {
#if COLLECT_PICK_MODE_STAT
- aom_usec_timer_start(&ms_stat.timer2);
+ aom_usec_timer_start(&ms_stat.timer2);
#endif
- av1_block_yrd(cpi, x, mi_row, mi_col, &this_rdc, &is_skippable, bsize,
- mi->tx_size, DCT_DCT, 1);
- if (this_rdc.skip_txfm ||
- RDCOST(x->rdmult, this_rdc.rate, this_rdc.dist) >=
- RDCOST(x->rdmult, 0, this_rdc.sse)) {
- if (!this_rdc.skip_txfm) {
- // Need to store "real" rdc for possible furure use if UV rdc
- // disallows tx skip
- nonskip_rdc = this_rdc;
- nonskip_rdc.rate += no_skip_txfm_cost;
- }
- this_rdc.rate = skip_txfm_cost;
- this_rdc.skip_txfm = 1;
- this_rdc.dist = this_rdc.sse;
- } else {
- this_rdc.rate += no_skip_txfm_cost;
+ av1_block_yrd(cpi, x, mi_row, mi_col, &this_rdc, &is_skippable, bsize,
+ mi->tx_size, DCT_DCT, 1);
+ if (this_rdc.skip_txfm ||
+ RDCOST(x->rdmult, this_rdc.rate, this_rdc.dist) >=
+ RDCOST(x->rdmult, 0, this_rdc.sse)) {
+ if (!this_rdc.skip_txfm) {
+ // Need to store "real" rdc for possible future use if UV rdc
+ // disallows tx skip
+ nonskip_rdc = this_rdc;
+ nonskip_rdc.rate += no_skip_txfm_cost;
}
+ this_rdc.rate = skip_txfm_cost;
+ this_rdc.skip_txfm = 1;
+ this_rdc.dist = this_rdc.sse;
+ } else {
+ this_rdc.rate += no_skip_txfm_cost;
}
if ((x->color_sensitivity[0] || x->color_sensitivity[1])) {
RD_STATS rdc_uv;
@@ -3294,6 +3371,9 @@
if (this_rdc.skip_txfm && !rdc_uv.skip_txfm &&
nonskip_rdc.rate != INT_MAX)
this_rdc = nonskip_rdc;
+ if (!comp_pred) {
+ uv_dist[INTER_OFFSET(this_mode)][ref_frame] = rdc_uv.dist;
+ }
this_rdc.rate += rdc_uv.rate;
this_rdc.dist += rdc_uv.dist;
this_rdc.skip_txfm = this_rdc.skip_txfm && rdc_uv.skip_txfm;
@@ -3329,7 +3409,7 @@
if (!comp_pred && frame_mv[this_mode][ref_frame].as_int == 0 &&
var < UINT_MAX) {
- zeromv_var[ref_frame] = var;
+ vars[INTER_OFFSET(GLOBALMV)][ref_frame] = var;
}
this_rdc.rate += ref_costs_single[ref_frame];
@@ -3375,7 +3455,7 @@
best_pickmode.best_mode_skip_txfm = this_rdc.skip_txfm;
best_pickmode.best_mode_initial_skip_flag =
(nonskip_rdc.rate == INT_MAX && this_rdc.skip_txfm);
- if (!best_pickmode.best_mode_skip_txfm && !use_modeled_non_rd_cost) {
+ if (!best_pickmode.best_mode_skip_txfm) {
memcpy(best_pickmode.blk_skip, txfm_info->blk_skip,
sizeof(txfm_info->blk_skip[0]) * num_8x8_blocks);
}
@@ -3428,16 +3508,24 @@
#if COLLECT_PICK_MODE_STAT
aom_usec_timer_start(&ms_stat.timer1);
ms_stat.num_searches[bsize][DC_PRED]++;
+ ms_stat.num_nonskipped_searches[bsize][DC_PRED]++;
#endif
- if (!x->force_zeromv_skip)
- estimate_intra_mode(cpi, x, bsize, use_modeled_non_rd_cost, best_early_term,
+ if (!x->force_zeromv_skip_for_blk)
+ estimate_intra_mode(cpi, x, bsize, best_early_term,
ref_costs_single[INTRA_FRAME], reuse_inter_pred,
&orig_dst, tmp, &this_mode_pred, &best_rdc,
&best_pickmode, ctx);
- if (cpi->oxcf.tune_cfg.content == AOM_CONTENT_SCREEN &&
- !cpi->oxcf.txfm_cfg.use_inter_dct_only && !x->force_zeromv_skip &&
+ int skip_idtx_palette =
+ (x->color_sensitivity[0] || x->color_sensitivity[1]) &&
+ x->content_state_sb.source_sad_nonrd != kZeroSad &&
+ !cpi->rc.high_source_sad;
+
+ // Check for IDTX: based only on Y channel, so avoid when color_sensitivity
+ // is set.
+ if (cpi->oxcf.tune_cfg.content == AOM_CONTENT_SCREEN && !skip_idtx_palette &&
+ !cpi->oxcf.txfm_cfg.use_inter_dct_only && !x->force_zeromv_skip_for_blk &&
is_inter_mode(best_pickmode.best_mode) &&
(!cpi->sf.rt_sf.prune_idtx_nonrd ||
(cpi->sf.rt_sf.prune_idtx_nonrd && bsize <= BLOCK_32X32 &&
@@ -3453,10 +3541,8 @@
mi->tx_size, IDTX, 1);
int64_t idx_rdcost = RDCOST(x->rdmult, idtx_rdc.rate, idtx_rdc.dist);
if (idx_rdcost < best_rdc.rdcost) {
- // Keep the skip_txfm off if the color_sensitivity is set,
- // for scene/slide change.
- if (cpi->rc.high_source_sad &&
- (x->color_sensitivity[0] || x->color_sensitivity[1]))
+ // Keep the skip_txfm off if the color_sensitivity is set.
+ if (x->color_sensitivity[0] || x->color_sensitivity[1])
idtx_rdc.skip_txfm = 0;
best_pickmode.tx_type = IDTX;
best_rdc.rdcost = idx_rdcost;
@@ -3473,11 +3559,11 @@
}
int try_palette =
- cpi->oxcf.tool_cfg.enable_palette &&
+ !skip_idtx_palette && cpi->oxcf.tool_cfg.enable_palette &&
av1_allow_palette(cpi->common.features.allow_screen_content_tools,
mi->bsize);
try_palette = try_palette && is_mode_intra(best_pickmode.best_mode) &&
- x->source_variance > 0 && !x->force_zeromv_skip &&
+ x->source_variance > 0 && !x->force_zeromv_skip_for_blk &&
(cpi->rc.high_source_sad || x->source_variance > 500);
if (try_palette) {
@@ -3493,6 +3579,9 @@
best_rdc.dist = this_rdc.dist;
best_rdc.rdcost = this_rdc.rdcost;
best_pickmode.best_mode_skip_txfm = this_rdc.skip_txfm;
+ // Keep the skip_txfm off if the color_sensitivity is set.
+ if (x->color_sensitivity[0] || x->color_sensitivity[1])
+ this_rdc.skip_txfm = 0;
if (!this_rdc.skip_txfm) {
memcpy(ctx->blk_skip, txfm_info->blk_skip,
sizeof(txfm_info->blk_skip[0]) * ctx->num_4x4_blk);
@@ -3583,52 +3672,64 @@
store_coding_context(x, ctx);
#endif // CONFIG_INTERNAL_STATS
#if COLLECT_PICK_MODE_STAT
- aom_usec_timer_mark(&ms_stat.timer2);
- ms_stat.avg_block_times[bsize] += aom_usec_timer_elapsed(&ms_stat.timer2);
- //
+ aom_usec_timer_mark(&ms_stat.bsize_timer);
+ ms_stat.total_block_times[bsize] +=
+ aom_usec_timer_elapsed(&ms_stat.bsize_timer);
if ((mi_row + mi_size_high[bsize] >= (cpi->common.mi_params.mi_rows)) &&
(mi_col + mi_size_wide[bsize] >= (cpi->common.mi_params.mi_cols))) {
- int i, j;
- BLOCK_SIZE bss[5] = { BLOCK_8X8, BLOCK_16X16, BLOCK_32X32, BLOCK_64X64,
- BLOCK_128X128 };
int64_t total_time = 0l;
int32_t total_blocks = 0;
+ for (BLOCK_SIZE bs = 0; bs < BLOCK_SIZES; bs++) {
+ total_time += ms_stat.total_block_times[bs];
+ total_blocks += ms_stat.num_blocks[bs];
+ }
printf("\n");
- for (i = 0; i < 5; i++) {
- printf("BS(%d) Num %d, Avg_time %f:\n", bss[i],
- ms_stat.num_blocks[bss[i]],
- ms_stat.num_blocks[bss[i]] > 0
- ? (float)ms_stat.avg_block_times[bss[i]] /
- ms_stat.num_blocks[bss[i]]
- : 0);
- total_time += ms_stat.avg_block_times[bss[i]];
- total_blocks += ms_stat.num_blocks[bss[i]];
- for (j = 0; j < MB_MODE_COUNT; j++) {
- if (ms_stat.nonskipped_search_times[bss[i]][j] == 0) {
+ for (BLOCK_SIZE bs = 0; bs < BLOCK_SIZES; bs++) {
+ if (ms_stat.num_blocks[bs] == 0) {
+ continue;
+ }
+ if (!COLLECT_NON_SQR_STAT && block_size_wide[bs] != block_size_high[bs]) {
+ continue;
+ }
+
+ printf("BLOCK_%dX%d Num %d, Time: %ld (%f%%), Avg_time %f:\n",
+ block_size_wide[bs], block_size_high[bs], ms_stat.num_blocks[bs],
+ ms_stat.total_block_times[bs],
+ 100 * ms_stat.total_block_times[bs] / (float)total_time,
+ (float)ms_stat.total_block_times[bs] / ms_stat.num_blocks[bs]);
+ for (int j = 0; j < MB_MODE_COUNT; j++) {
+ if (ms_stat.nonskipped_search_times[bs][j] == 0) {
continue;
}
+ int64_t total_mode_time = ms_stat.nonskipped_search_times[bs][j];
printf(" Mode %d, %d/%d tps %f\n", j,
- ms_stat.num_nonskipped_searches[bss[i]][j],
- ms_stat.num_searches[bss[i]][j],
- ms_stat.num_nonskipped_searches[bss[i]][j] > 0
- ? (float)ms_stat.nonskipped_search_times[bss[i]][j] /
- ms_stat.num_nonskipped_searches[bss[i]][j]
+ ms_stat.num_nonskipped_searches[bs][j],
+ ms_stat.num_searches[bs][j],
+ ms_stat.num_nonskipped_searches[bs][j] > 0
+ ? (float)ms_stat.nonskipped_search_times[bs][j] /
+ ms_stat.num_nonskipped_searches[bs][j]
: 0l);
if (j >= INTER_MODE_START) {
- printf(" Motion Search Time: %ld\n", ms_stat.ms_time[bss[i]][j]);
- printf(" Filter Search Time: %ld\n", ms_stat.ifs_time[bss[i]][j]);
- printf(" Model RD Time: %ld\n",
- ms_stat.model_rd_time[bss[i]][j]);
- printf(" Tranfm Search Time: %ld\n", ms_stat.txfm_time[bss[i]][j]);
+ total_mode_time = ms_stat.ms_time[bs][j] + ms_stat.ifs_time[bs][j] +
+ ms_stat.model_rd_time[bs][j] +
+ ms_stat.txfm_time[bs][j];
+ print_stage_time("Motion Search Time", ms_stat.ms_time[bs][j],
+ total_time);
+ print_stage_time("Filter Search Time", ms_stat.ifs_time[bs][j],
+ total_time);
+ print_stage_time("Model RD Time", ms_stat.model_rd_time[bs][j],
+ total_time);
+ print_stage_time("Tranfm Search Time", ms_stat.txfm_time[bs][j],
+ total_time);
}
+ print_stage_time("Total Mode Time", total_mode_time, total_time);
}
printf("\n");
}
printf("Total time = %ld. Total blocks = %d\n", total_time, total_blocks);
}
- //
#endif // COLLECT_PICK_MODE_STAT
*rd_cost = best_rdc;
}
diff --git a/av1/encoder/palette.c b/av1/encoder/palette.c
index 3363336..4375175 100644
--- a/av1/encoder/palette.c
+++ b/av1/encoder/palette.c
@@ -219,7 +219,7 @@
BLOCK_SIZE bsize, int dc_mode_cost, const int *data, int *centroids, int n,
uint16_t *color_cache, int n_cache, bool do_header_rd_based_gating,
MB_MODE_INFO *best_mbmi, uint8_t *best_palette_color_map, int64_t *best_rd,
- int *rate, int *rate_tokenonly, int64_t *distortion, int *skippable,
+ int *rate, int *rate_tokenonly, int64_t *distortion, uint8_t *skippable,
int *beat_best_rd, PICK_MODE_CONTEXT *ctx, uint8_t *blk_skip,
uint8_t *tx_type_map, int *beat_best_palette_rd,
bool *do_header_rd_based_breakout, int discount_color_cost) {
@@ -328,7 +328,7 @@
int start_n, int end_n, int step_size, bool do_header_rd_based_gating,
int *last_n_searched, uint16_t *color_cache, int n_cache,
MB_MODE_INFO *best_mbmi, uint8_t *best_palette_color_map, int64_t *best_rd,
- int *rate, int *rate_tokenonly, int64_t *distortion, int *skippable,
+ int *rate, int *rate_tokenonly, int64_t *distortion, uint8_t *skippable,
int *beat_best_rd, PICK_MODE_CONTEXT *ctx, uint8_t *best_blk_skip,
uint8_t *tx_type_map, int discount_color_cost) {
int centroids[PALETTE_MAX_SIZE];
@@ -376,7 +376,7 @@
bool do_header_rd_based_gating, int *last_n_searched, uint16_t *color_cache,
int n_cache, MB_MODE_INFO *best_mbmi, uint8_t *best_palette_color_map,
int64_t *best_rd, int *rate, int *rate_tokenonly, int64_t *distortion,
- int *skippable, int *beat_best_rd, PICK_MODE_CONTEXT *ctx,
+ uint8_t *skippable, int *beat_best_rd, PICK_MODE_CONTEXT *ctx,
uint8_t *best_blk_skip, uint8_t *tx_type_map, uint8_t *color_map,
int data_points, int discount_color_cost) {
int centroids[PALETTE_MAX_SIZE];
@@ -527,7 +527,7 @@
void av1_rd_pick_palette_intra_sby(
const AV1_COMP *cpi, MACROBLOCK *x, BLOCK_SIZE bsize, int dc_mode_cost,
MB_MODE_INFO *best_mbmi, uint8_t *best_palette_color_map, int64_t *best_rd,
- int *rate, int *rate_tokenonly, int64_t *distortion, int *skippable,
+ int *rate, int *rate_tokenonly, int64_t *distortion, uint8_t *skippable,
int *beat_best_rd, PICK_MODE_CONTEXT *ctx, uint8_t *best_blk_skip,
uint8_t *tx_type_map) {
MACROBLOCKD *const xd = &x->e_mbd;
@@ -740,7 +740,7 @@
MB_MODE_INFO *const best_mbmi,
int64_t *best_rd, int *rate,
int *rate_tokenonly, int64_t *distortion,
- int *skippable) {
+ uint8_t *skippable) {
MACROBLOCKD *const xd = &x->e_mbd;
MB_MODE_INFO *const mbmi = xd->mi[0];
assert(!is_inter_block(mbmi));
diff --git a/av1/encoder/palette.h b/av1/encoder/palette.h
index 7d9a72f..34d2ddc 100644
--- a/av1/encoder/palette.h
+++ b/av1/encoder/palette.h
@@ -49,7 +49,7 @@
* \param[in] k Number of clusters.
* \param[in] dim Data dimension.
*
- * \return Returns nothing, but saves each data's cluster index in indices.
+ * \remark Returns nothing, but saves each data's cluster index in \a indices.
*/
static INLINE void av1_calc_indices(const int *data, const int *centroids,
uint8_t *indices, int n, int k, int dim) {
@@ -79,8 +79,8 @@
* \param[in] dim Data dimension.
* \param[in] max_itr Maximum number of iterations to run.
*
- * \return Returns nothing, but saves each cluster's centroid in centroids and
- * each data's cluster index in indices.
+ * \remark Returns nothing, but saves each cluster's centroid in centroids and
+ * each data's cluster index in \a indices.
*
* \attention The output centroids are rounded off to nearest integers.
*/
@@ -186,7 +186,7 @@
const struct AV1_COMP *cpi, struct macroblock *x, BLOCK_SIZE bsize,
int dc_mode_cost, MB_MODE_INFO *best_mbmi, uint8_t *best_palette_color_map,
int64_t *best_rd, int *rate, int *rate_tokenonly, int64_t *distortion,
- int *skippable, int *beat_best_rd, struct PICK_MODE_CONTEXT *ctx,
+ uint8_t *skippable, int *beat_best_rd, struct PICK_MODE_CONTEXT *ctx,
uint8_t *best_blk_skip, uint8_t *tx_type_map);
/*!\brief Search for the best palette in the chroma plane.
@@ -201,7 +201,7 @@
MB_MODE_INFO *const best_mbmi,
int64_t *best_rd, int *rate,
int *rate_tokenonly, int64_t *distortion,
- int *skippable);
+ uint8_t *skippable);
/*!\brief Resets palette color map for chroma channels.
*/
diff --git a/av1/encoder/partition_search.c b/av1/encoder/partition_search.c
index 4dd47e0..af7229a 100644
--- a/av1/encoder/partition_search.c
+++ b/av1/encoder/partition_search.c
@@ -724,7 +724,7 @@
* \param[in] ctx Structure to hold snapshot of coding context
during the mode picking process
*
- * \return Nothing is returned. Instead, the MB_MODE_INFO struct inside x
+ * \remark Nothing is returned. Instead, the MB_MODE_INFO struct inside x
* is modified to store information about the best mode computed
* in this function. The rd_cost struct is also updated with the RD stats
* corresponding to the best mode found.
@@ -751,18 +751,9 @@
av1_nonrd_pick_intra_mode(cpi, x, rd_cost, bsize, ctx);
}
-// For real time/allintra row-mt enabled multi-threaded encoding with cost
-// update frequency set to COST_UPD_TILE/COST_UPD_OFF, tile ctxt is not updated
-// at superblock level. Thus, it is not required for the encoding of top-right
-// superblock be complete for updating tile ctxt. However, when encoding a block
-// whose right edge is also the superblock edge, intra and inter mode evaluation
-// (ref mv list population) require the encoding of the top-right superblock to
-// be complete. So, here, we delay the waiting of threads until the need for the
-// data from the top-right superblock region.
-static AOM_INLINE void wait_for_top_right_sb(
- AV1EncRowMultiThreadInfo *enc_row_mt, AV1EncRowMultiThreadSync *row_mt_sync,
- TileInfo *tile_info, BLOCK_SIZE sb_size, int sb_mi_size_log2,
- BLOCK_SIZE bsize, int mi_row, int mi_col) {
+static AOM_INLINE int is_top_right_block_in_sb(BLOCK_SIZE sb_size,
+ BLOCK_SIZE bsize, int mi_row,
+ int mi_col) {
const int sb_size_in_mi = mi_size_wide[sb_size];
const int bw_in_mi = mi_size_wide[bsize];
const int blk_row_in_sb = mi_row & (sb_size_in_mi - 1);
@@ -770,16 +761,103 @@
const int top_right_block_in_sb =
(blk_row_in_sb == 0) && (blk_col_in_sb + bw_in_mi >= sb_size_in_mi);
- // Don't wait if the block is the not the top-right block in the superblock.
- if (!top_right_block_in_sb) return;
+ return top_right_block_in_sb;
+}
- // Wait for the top-right superblock to finish encoding.
+// For real time/allintra row-mt enabled multi-threaded encoding with cost
+// update frequency set to COST_UPD_TILE/COST_UPD_OFF, tile ctxt is not updated
+// at superblock level. Thus, it is not required for the encoding of top-right
+// superblock be complete for updating tile ctxt. However, when encoding a block
+// whose right edge is also the superblock edge, intra and inter mode evaluation
+// (ref mv list population) require the encoding of the top-right region to
+// be complete. So, here, we delay the waiting of threads until the need for the
+// data from the top-right superblock region.
+static AOM_INLINE void wait_for_top_right(AV1_COMP *cpi,
+ AV1EncRowMultiThreadSync *row_mt_sync,
+ TileInfo *tile_info,
+ BLOCK_SIZE sb_size, BLOCK_SIZE bsize,
+ int mi_row, int mi_col,
+ int seg_skip_active) {
+ // Don't wait if the block is the not the top-right block in the superblock.
+ if (!is_top_right_block_in_sb(sb_size, bsize, mi_row, mi_col)) return;
+
+ AV1EncRowMultiThreadInfo *enc_row_mt = &cpi->mt_info.enc_row_mt;
+ const int sb_mi_size_log2 = mi_size_wide_log2[sb_size];
const int sb_row_in_tile =
(mi_row - tile_info->mi_row_start) >> sb_mi_size_log2;
- const int sb_col_in_tile =
- (mi_col - tile_info->mi_col_start) >> sb_mi_size_log2;
- enc_row_mt->sync_read_ptr(row_mt_sync, sb_row_in_tile, sb_col_in_tile);
+ // In case of non-rd RT with row-mt enabled, encoding of SB can start after
+ // encoding of bottom left block in above right superblock is complete. This
+ // is because only DC, H and V intra modes are enabled via the speed feature
+ // intra_y_mode_bsize_mask_nrd (above right region not required) and reference
+ // mv list population requires only the above right block info.
+ if (enable_top_right_sync_wait_in_mis(cpi, seg_skip_active)) {
+ const int *intra_y_mode_bsize_mask_nrd =
+ cpi->sf.rt_sf.intra_y_mode_bsize_mask_nrd;
+ for (int i = 0; i < BLOCK_SIZES; ++i)
+ assert(intra_y_mode_bsize_mask_nrd[i] == INTRA_DC ||
+ intra_y_mode_bsize_mask_nrd[i] == INTRA_DC_H_V);
+ (void)intra_y_mode_bsize_mask_nrd;
+#if CONFIG_MULTITHREAD
+ const int mi_col_in_tile = mi_col - tile_info->mi_col_start;
+ const int mi_cols_in_tile = tile_info->mi_col_end - tile_info->mi_col_start;
+ const int bw_in_mi = mi_size_wide[bsize];
+ if (sb_row_in_tile) {
+ pthread_mutex_t *const mutex = &row_mt_sync->mutex_[sb_row_in_tile - 1];
+ pthread_mutex_lock(mutex);
+
+ while (AOMMIN(mi_col_in_tile + bw_in_mi, mi_cols_in_tile) >=
+ row_mt_sync->finished_block_in_mi[sb_row_in_tile - 1]) {
+ pthread_cond_wait(&row_mt_sync->cond_[sb_row_in_tile - 1], mutex);
+ }
+ pthread_mutex_unlock(mutex);
+ }
+#endif
+ } else {
+ const int sb_col_in_tile =
+ (mi_col - tile_info->mi_col_start) >> sb_mi_size_log2;
+ enc_row_mt->sync_read_ptr(row_mt_sync, sb_row_in_tile, sb_col_in_tile);
+ }
+}
+
+static AOM_INLINE void write_completed_mi_pos(
+ AV1EncRowMultiThreadSync *row_mt_sync, TileInfo *tile_info,
+ BLOCK_SIZE sb_size, BLOCK_SIZE bsize, int mi_row, int mi_col) {
+ const int sb_size_in_mi = mi_size_high[sb_size];
+ const int bh_in_mi = mi_size_high[bsize];
+ const int blk_row_in_sb = mi_row & (sb_size_in_mi - 1);
+ const int bottom_block_in_sb = blk_row_in_sb + bh_in_mi >= sb_size_in_mi;
+
+ // Don't write if the block is the not the bottom block in the
+ // superblock.
+ if (!bottom_block_in_sb) return;
+
+#if CONFIG_MULTITHREAD
+ const int sb_mi_size_log2 = mi_size_wide_log2[sb_size];
+ const int sb_row_in_tile =
+ (mi_row - tile_info->mi_row_start) >> sb_mi_size_log2;
+ const int bw_in_mi = mi_size_wide[bsize];
+ const int mi_col_in_tile = mi_col + bw_in_mi - tile_info->mi_col_start;
+ const int mi_cols_in_tile = tile_info->mi_col_end - tile_info->mi_col_start;
+
+ const int finished_mi_col = mi_col_in_tile < mi_cols_in_tile - 1
+ ? mi_col_in_tile
+ : mi_cols_in_tile + 1;
+
+ pthread_mutex_lock(&row_mt_sync->mutex_[sb_row_in_tile]);
+
+ row_mt_sync->finished_block_in_mi[sb_row_in_tile] = finished_mi_col;
+
+ pthread_cond_signal(&row_mt_sync->cond_[sb_row_in_tile]);
+ pthread_mutex_unlock(&row_mt_sync->mutex_[sb_row_in_tile]);
+#else
+ (void)row_mt_sync;
+ (void)tile_info;
+ (void)sb_size;
+ (void)bsize;
+ (void)mi_row;
+ (void)mi_col;
+#endif // CONFIG_MULTITHREAD
}
/*!\brief Interface for AV1 mode search for an individual coding block
@@ -810,7 +888,7 @@
* chosen modes for the current block
* \param[in] best_rd Upper bound of rd cost of a valid partition
*
- * \return Nothing is returned. Instead, the chosen modes and contexts necessary
+ * \remark Nothing is returned. Instead, the chosen modes and contexts necessary
* for reconstruction are stored in ctx, the rate-distortion stats are stored in
* rd_cost. If no valid mode leading to rd_cost <= best_rd, the status will be
* signalled by an INT64_MAX rd_cost->rdcost.
@@ -852,9 +930,8 @@
// This is only needed for real time/allintra row-mt enabled multi-threaded
// encoding with cost update frequency set to COST_UPD_TILE/COST_UPD_OFF.
- wait_for_top_right_sb(&cpi->mt_info.enc_row_mt, &tile_data->row_mt_sync,
- &tile_data->tile_info, cm->seq_params->sb_size,
- cm->seq_params->mib_size_log2, bsize, mi_row, mi_col);
+ wait_for_top_right(cpi, &tile_data->row_mt_sync, &tile_data->tile_info,
+ cm->seq_params->sb_size, bsize, mi_row, mi_col, 0);
#if CONFIG_COLLECT_COMPONENT_TIMING
start_timing(cpi, rd_pick_sb_modes_time);
@@ -1391,7 +1468,7 @@
* chosen modes for the current block
* \param[in] rate Pointer to the total rate for the current block
*
- * \return Nothing is returned. Instead, reconstructions (w/o in-loop filters)
+ * \remark Nothing is returned. Instead, reconstructions (w/o in-loop filters)
* will be updated in the pixel buffers in td->mb.e_mbd. Also, the chosen modes
* will be stored in the MB_MODE_INFO buffer td->mb.e_mbd.mi[0].
*/
@@ -1554,7 +1631,7 @@
* partitions and mode info for the current block
* \param[in] rate Pointer to the total rate for the current block
*
- * \return Nothing is returned. Instead, reconstructions (w/o in-loop filters)
+ * \remark Nothing is returned. Instead, reconstructions (w/o in-loop filters)
* will be updated in the pixel buffers in td->mb.e_mbd.
*/
static void encode_sb(const AV1_COMP *const cpi, ThreadData *td,
@@ -1736,7 +1813,7 @@
* \param[in] pc_tree Pointer to the PC_TREE node holding the picked
partitions and mode info for the current block
*
-* \return Nothing is returned. The pc_tree struct is modified to store the
+* \remark Nothing is returned. The pc_tree struct is modified to store the
* picked partition and modes. The rate and dist are also updated with those
* corresponding to the best partition found.
*/
@@ -2139,6 +2216,51 @@
#if CONFIG_COLLECT_COMPONENT_TIMING
end_timing((AV1_COMP *)cpi, encode_b_nonrd_time);
#endif
+ const int seg_skip_active =
+ segfeature_active(&cm->seg, mbmi->segment_id, SEG_LVL_SKIP);
+ if (!dry_run && enable_top_right_sync_wait_in_mis(cpi, seg_skip_active))
+ write_completed_mi_pos(&tile_data->row_mt_sync, &tile_data->tile_info,
+ cm->seq_params->sb_size, bsize, mi_row, mi_col);
+}
+
+static int get_force_zeromv_skip_flag_for_blk(const AV1_COMP *cpi,
+ const MACROBLOCK *x,
+ BLOCK_SIZE bsize) {
+ // Force zero MV skip based on SB level decision
+ if (x->force_zeromv_skip_for_sb < 2) return x->force_zeromv_skip_for_sb;
+
+ // For blocks of size equal to superblock size, the decision would have been
+ // already done at superblock level. Hence zeromv-skip decision is skipped.
+ const AV1_COMMON *const cm = &cpi->common;
+ if (bsize == cm->seq_params->sb_size) return 0;
+
+ const int num_planes = av1_num_planes(cm);
+ const MACROBLOCKD *const xd = &x->e_mbd;
+ const unsigned int thresh_exit_part_y =
+ cpi->zeromv_skip_thresh_exit_part[bsize];
+ const unsigned int thresh_exit_part_uv =
+ CALC_CHROMA_THRESH_FOR_ZEROMV_SKIP(thresh_exit_part_y);
+ const unsigned int thresh_exit_part[MAX_MB_PLANE] = { thresh_exit_part_y,
+ thresh_exit_part_uv,
+ thresh_exit_part_uv };
+ const YV12_BUFFER_CONFIG *const yv12 = get_ref_frame_yv12_buf(cm, LAST_FRAME);
+ const struct scale_factors *const sf =
+ get_ref_scale_factors_const(cm, LAST_FRAME);
+
+ struct buf_2d yv12_mb[MAX_MB_PLANE];
+ av1_setup_pred_block(xd, yv12_mb, yv12, sf, sf, num_planes);
+
+ for (int plane = 0; plane < num_planes; ++plane) {
+ const struct macroblock_plane *const p = &x->plane[plane];
+ const struct macroblockd_plane *const pd = &xd->plane[plane];
+ const BLOCK_SIZE bs =
+ get_plane_block_size(bsize, pd->subsampling_x, pd->subsampling_y);
+ const unsigned int plane_sad = cpi->ppi->fn_ptr[bs].sdf(
+ p->src.buf, p->src.stride, yv12_mb[plane].buf, yv12_mb[plane].stride);
+ assert(plane < MAX_MB_PLANE);
+ if (plane_sad >= thresh_exit_part[plane]) return 0;
+ }
+ return 1;
}
/*!\brief Top level function to pick block mode for non-RD optimized case
@@ -2168,7 +2290,7 @@
* \param[in] ctx Pointer to structure holding coding contexts and
* chosen modes for the current block
*
- * \return Nothing is returned. Instead, the chosen modes and contexts necessary
+ * \remark Nothing is returned. Instead, the chosen modes and contexts necessary
* for reconstruction are stored in ctx, the rate-distortion stats are stored in
* rd_cost. If no valid mode leading to rd_cost <= best_rd, the status will be
* signalled by an INT64_MAX rd_cost->rdcost.
@@ -2195,12 +2317,6 @@
TxfmSearchInfo *txfm_info = &x->txfm_search_info;
int i;
- // This is only needed for real time/allintra row-mt enabled multi-threaded
- // encoding with cost update frequency set to COST_UPD_TILE/COST_UPD_OFF.
- wait_for_top_right_sb(&cpi->mt_info.enc_row_mt, &tile_data->row_mt_sync,
- &tile_data->tile_info, cm->seq_params->sb_size,
- cm->seq_params->mib_size_log2, bsize, mi_row, mi_col);
-
#if CONFIG_COLLECT_COMPONENT_TIMING
start_timing(cpi, pick_sb_modes_nonrd_time);
#endif
@@ -2215,7 +2331,11 @@
p[i].txb_entropy_ctx = ctx->txb_entropy_ctx[i];
}
for (i = 0; i < 2; ++i) pd[i].color_index_map = ctx->color_index_map[i];
- if (!x->force_zeromv_skip) {
+
+ x->force_zeromv_skip_for_blk =
+ get_force_zeromv_skip_flag_for_blk(cpi, x, bsize);
+
+ if (!x->force_zeromv_skip_for_blk) {
x->source_variance = av1_get_perpixel_variance_facade(
cpi, xd, &x->plane[0].src, bsize, AOM_PLANE_Y);
}
@@ -2224,6 +2344,15 @@
setup_block_rdmult(cpi, x, mi_row, mi_col, bsize, aq_mode, mbmi);
// Set error per bit for current rdmult
av1_set_error_per_bit(&x->errorperbit, x->rdmult);
+
+ const int seg_skip_active =
+ segfeature_active(&cm->seg, mbmi->segment_id, SEG_LVL_SKIP);
+ // This is only needed for real time/allintra row-mt enabled multi-threaded
+ // encoding with cost update frequency set to COST_UPD_TILE/COST_UPD_OFF.
+ wait_for_top_right(cpi, &tile_data->row_mt_sync, &tile_data->tile_info,
+ cm->seq_params->sb_size, bsize, mi_row, mi_col,
+ seg_skip_active);
+
// Find best coding mode & reconstruct the MB so it is available
// as a predictor for MBs that follow in the SB
if (frame_is_intra_only(cm)) {
@@ -2238,7 +2367,7 @@
#if CONFIG_COLLECT_COMPONENT_TIMING
start_timing(cpi, nonrd_pick_inter_mode_sb_time);
#endif
- if (segfeature_active(&cm->seg, mbmi->segment_id, SEG_LVL_SKIP)) {
+ if (seg_skip_active) {
RD_STATS invalid_rd;
av1_invalid_rd_stats(&invalid_rd);
// TODO(kyslov): add av1_nonrd_pick_inter_mode_sb_seg_skip
@@ -2253,6 +2382,12 @@
#endif
}
if (cpi->sf.rt_sf.skip_cdef_sb) {
+ // cdef_strength is initialized to 1 which means skip_cdef, and is updated
+ // here. Check to see is skipping cdef is allowed.
+ const int allow_cdef_skipping =
+ cpi->rc.frames_since_key > 10 && !cpi->rc.high_source_sad &&
+ !(x->color_sensitivity[0] || x->color_sensitivity[1]);
+
// Find the corresponding 64x64 block. It'll be the 128x128 block if that's
// the block size.
const int mi_row_sb = mi_row - mi_row % MI_SIZE_64X64;
@@ -2262,12 +2397,11 @@
get_mi_grid_idx(&cm->mi_params, mi_row_sb, mi_col_sb);
// Do not skip if intra or new mv is picked, or color sensitivity is set.
// Never skip on slide/scene change.
- mi_sb[0]->skip_cdef_curr_sb =
- mi_sb[0]->skip_cdef_curr_sb && !cpi->rc.high_source_sad &&
- !(x->color_sensitivity[0] || x->color_sensitivity[1]) &&
+ mi_sb[0]->cdef_strength =
+ mi_sb[0]->cdef_strength && allow_cdef_skipping &&
!(mbmi->mode < INTRA_MODES || mbmi->mode == NEWMV);
// Store in the pickmode context.
- ctx->mic.skip_cdef_curr_sb = mi_sb[0]->skip_cdef_curr_sb;
+ ctx->mic.cdef_strength = mi_sb[0]->cdef_strength;
}
x->rdmult = orig_rdmult;
ctx->rd_stats.rate = rd_cost->rate;
@@ -2495,7 +2629,7 @@
* \param[in] pc_tree Pointer to the PC_TREE node holding the picked
partitions and mode info for the current block
*
-* \return Nothing is returned. The pc_tree struct is modified to store the
+* \remark Nothing is returned. The pc_tree struct is modified to store the
* picked partition and modes.
*/
void av1_nonrd_use_partition(AV1_COMP *cpi, ThreadData *td,
diff --git a/av1/encoder/partition_strategy.c b/av1/encoder/partition_strategy.c
index f66fd08..89c1a79 100644
--- a/av1/encoder/partition_strategy.c
+++ b/av1/encoder/partition_strategy.c
@@ -1767,11 +1767,10 @@
// Threshold for number of winners
// Conservative pruning for high quantizers
const int num_win_thresh = AOMMIN(3 * (2 * (MAXQ - qindex) / MAXQ), 3);
- int sub_part_win = (rect_part_win_info == NULL)
- ? (pc_tree->partitioning == rect_part)
- : (rect_part == PARTITION_HORZ)
- ? rect_part_win_info->rect_part_win[HORZ]
- : rect_part_win_info->rect_part_win[VERT];
+ int sub_part_win =
+ (rect_part_win_info == NULL) ? (pc_tree->partitioning == rect_part)
+ : (rect_part == PARTITION_HORZ) ? rect_part_win_info->rect_part_win[HORZ]
+ : rect_part_win_info->rect_part_win[VERT];
num_win += (sub_part_win) ? 1 : 0;
if (pc_tree->split[split_idx1]) {
num_win +=
diff --git a/av1/encoder/pass2_strategy.c b/av1/encoder/pass2_strategy.c
index 0dd0939..256365a 100644
--- a/av1/encoder/pass2_strategy.c
+++ b/av1/encoder/pass2_strategy.c
@@ -799,11 +799,10 @@
}
// Clamp odd edge cases.
- total_group_bits = (total_group_bits < 0)
- ? 0
- : (total_group_bits > twopass->kf_group_bits)
- ? twopass->kf_group_bits
- : total_group_bits;
+ total_group_bits = (total_group_bits < 0) ? 0
+ : (total_group_bits > twopass->kf_group_bits)
+ ? twopass->kf_group_bits
+ : total_group_bits;
// Clip based on user supplied data rate variability limit.
if (total_group_bits > (int64_t)max_bits * p_rc->baseline_gf_interval)
@@ -1876,7 +1875,7 @@
* \param[in] max_gop_length Maximum length of the GF group
* \param[in] max_intervals Maximum number of intervals to decide
*
- * \return Nothing is returned. Instead, cpi->ppi->rc.gf_intervals is
+ * \remark Nothing is returned. Instead, cpi->ppi->rc.gf_intervals is
* changed to store the decided GF group lengths.
*/
static void calculate_gf_length(AV1_COMP *cpi, int max_gop_length,
@@ -2143,7 +2142,7 @@
*
* \param[in] cpi Top-level encoder structure
*
- * \return Nothing is returned. Instead, cpi->ppi->gf_group is changed.
+ * \remark Nothing is returned. Instead, cpi->ppi->gf_group is changed.
*/
static void define_gf_group_pass0(AV1_COMP *cpi) {
RATE_CONTROL *const rc = &cpi->rc;
@@ -2432,7 +2431,7 @@
* \param[in] is_final_pass Whether this is the final pass for the
* GF group, or a trial (non-zero)
*
- * \return Nothing is returned. Instead, cpi->ppi->gf_group is changed.
+ * \remark Nothing is returned. Instead, cpi->ppi->gf_group is changed.
*/
static void define_gf_group(AV1_COMP *cpi, EncodeFrameParams *frame_params,
int is_final_pass) {
@@ -3147,8 +3146,6 @@
*
* \param[in] cpi Top-level encoder structure
* \param[in] this_frame Pointer to first pass stats
- *
- * \return Nothing is returned.
*/
static void find_next_key_frame(AV1_COMP *cpi, FIRSTPASS_STATS *this_frame) {
RATE_CONTROL *const rc = &cpi->rc;
diff --git a/av1/encoder/pass2_strategy.h b/av1/encoder/pass2_strategy.h
index 6234623..87e08bb 100644
--- a/av1/encoder/pass2_strategy.h
+++ b/av1/encoder/pass2_strategy.h
@@ -21,7 +21,6 @@
#include "av1/encoder/encoder.h"
-/*!\endcond */
/*!
* \brief accumulated stats and features in a gf group
*/
@@ -60,7 +59,7 @@
double frame_sr_coded_error;
/*!\endcond */
} GF_FRAME_STATS;
-/*!cond */
+/*!\cond */
void av1_init_second_pass(struct AV1_COMP *cpi);
@@ -83,7 +82,7 @@
* \param[in] frame_params Per frame encoding parameters
* \param[in] frame_flags Frame type and coding flags
*
- * \return No return but analyses first pass stats and assigns a target
+ * \remark No return but analyses first pass stats and assigns a target
* number of bits to the current frame and a target Q range.
*/
void av1_get_second_pass_params(struct AV1_COMP *cpi,
@@ -99,7 +98,7 @@
*
* \param[in] cpi Top - level encoder instance structure
*
- * \return No return value but this function updates various rate control
+ * \remark No return value but this function updates various rate control
* related data structures that for example track overshoot and
* undershoot.
*/
@@ -121,7 +120,7 @@
* uni-directional group.
* \param[in] gf_group_bits Bits available to be allocated.
*
- * \return No return but updates the rate control and group data structures
+ * \remark No return but updates the rate control and group data structures
* to reflect the allocation of bits.
*/
void av1_gop_bit_allocation(const AV1_COMP *cpi, RATE_CONTROL *const rc,
diff --git a/av1/encoder/pickcdef.c b/av1/encoder/pickcdef.c
index adbb07c..3659650 100644
--- a/av1/encoder/pickcdef.c
+++ b/av1/encoder/pickcdef.c
@@ -508,7 +508,7 @@
}
static void pick_cdef_from_qp(AV1_COMMON *const cm, int skip_cdef,
- int frames_since_key, int is_screen_content) {
+ int is_screen_content) {
const int bd = cm->seq_params->bit_depth;
const int q =
av1_ac_quant_QTX(cm->quant_params.base_qindex, 0, bd) >> (bd - 8);
@@ -574,10 +574,14 @@
cdef_info->cdef_uv_strengths[0] =
predicted_uv_f1 * CDEF_SEC_STRENGTHS + predicted_uv_f2;
+ // mbmi->cdef_strength is already set in the encoding stage. We don't need to
+ // set it again here.
if (skip_cdef) {
cdef_info->cdef_strengths[1] = 0;
cdef_info->cdef_uv_strengths[1] = 0;
+ return;
}
+
const CommonModeInfoParams *const mi_params = &cm->mi_params;
const int nvfb = (mi_params->mi_rows + MI_SIZE_64X64 - 1) / MI_SIZE_64X64;
const int nhfb = (mi_params->mi_cols + MI_SIZE_64X64 - 1) / MI_SIZE_64X64;
@@ -586,10 +590,6 @@
for (int c = 0; c < nhfb; ++c) {
MB_MODE_INFO *current_mbmi = mbmi[MI_SIZE_64X64 * c];
current_mbmi->cdef_strength = 0;
- if (skip_cdef && current_mbmi->skip_cdef_curr_sb &&
- frames_since_key > 10) {
- current_mbmi->cdef_strength = 1;
- }
}
mbmi += MI_SIZE_64X64 * mi_params->mi_stride;
}
@@ -598,9 +598,8 @@
void av1_cdef_search(MultiThreadInfo *mt_info, const YV12_BUFFER_CONFIG *frame,
const YV12_BUFFER_CONFIG *ref, AV1_COMMON *cm,
MACROBLOCKD *xd, CDEF_PICK_METHOD pick_method, int rdmult,
- int skip_cdef_feature, int frames_since_key,
- CDEF_CONTROL cdef_control, const int is_screen_content,
- int non_reference_frame) {
+ int skip_cdef_feature, CDEF_CONTROL cdef_control,
+ const int is_screen_content, int non_reference_frame) {
assert(cdef_control != CDEF_NONE);
if (cdef_control == CDEF_REFERENCE && non_reference_frame) {
CdefInfo *const cdef_info = &cm->cdef_info;
@@ -612,8 +611,7 @@
}
if (pick_method == CDEF_PICK_FROM_Q) {
- pick_cdef_from_qp(cm, skip_cdef_feature, frames_since_key,
- is_screen_content);
+ pick_cdef_from_qp(cm, skip_cdef_feature, is_screen_content);
return;
}
const CommonModeInfoParams *const mi_params = &cm->mi_params;
diff --git a/av1/encoder/pickcdef.h b/av1/encoder/pickcdef.h
index e070a8a..077edcb 100644
--- a/av1/encoder/pickcdef.h
+++ b/av1/encoder/pickcdef.h
@@ -226,13 +226,12 @@
* \param[in] pick_method The method used to select params
* \param[in] rdmult rd multiplier to use in making param choices
* \param[in] skip_cdef_feature Speed feature to skip cdef
- * \param[in] frames_since_key Number of frames since key frame
* \param[in] cdef_control Parameter that controls CDEF application
* \param[in] is_screen_content Whether it is screen content type
* \param[in] non_reference_frame Indicates if current frame is
* non-reference
*
- * \return Nothing is returned. Instead, optimal CDEF parameters are stored
+ * \remark Nothing is returned. Instead, optimal CDEF parameters are stored
* in the \c cdef_info structure of type \ref CdefInfo inside \c cm:
* \arg \c cdef_bits: Bits of strength parameters
* \arg \c nb_cdef_strengths: Number of strength parameters
@@ -247,9 +246,8 @@
const YV12_BUFFER_CONFIG *frame,
const YV12_BUFFER_CONFIG *ref, AV1_COMMON *cm,
MACROBLOCKD *xd, CDEF_PICK_METHOD pick_method, int rdmult,
- int skip_cdef_feature, int frames_since_key,
- CDEF_CONTROL cdef_control, const int is_screen_content,
- int non_reference_frame);
+ int skip_cdef_feature, CDEF_CONTROL cdef_control,
+ const int is_screen_content, int non_reference_frame);
#ifdef __cplusplus
} // extern "C"
diff --git a/av1/encoder/picklpf.c b/av1/encoder/picklpf.c
index 3d3020a..47d007e 100644
--- a/av1/encoder/picklpf.c
+++ b/av1/encoder/picklpf.c
@@ -212,7 +212,7 @@
if (disable_filter_rt_screen ||
cpi->oxcf.algo_cfg.loopfilter_control == LOOPFILTER_NONE ||
(cpi->oxcf.algo_cfg.loopfilter_control == LOOPFILTER_REFERENCE &&
- cpi->svc.non_reference_frame)) {
+ cpi->rtc_ref.non_reference_frame)) {
lf->filter_level[0] = 0;
lf->filter_level[1] = 0;
return;
diff --git a/av1/encoder/picklpf.h b/av1/encoder/picklpf.h
index 7273355..f567937 100644
--- a/av1/encoder/picklpf.h
+++ b/av1/encoder/picklpf.h
@@ -43,7 +43,7 @@
* \arg \c LPF_PICK_MINIMAL_LPF: Pick 0 to disable LPF if LPF was enabled last
* frame
*
- * \return Nothing is returned. Instead, filter levels below are stored in the
+ * \remark Nothing is returned. Instead, filter levels below are stored in the
* "loopfilter" structure inside "cpi":
* \arg \c filter_level[0]: the vertical filter level for Y plane
* \arg \c filter_level[1]: the horizontal filter level for Y plane
diff --git a/av1/encoder/pickrst.h b/av1/encoder/pickrst.h
index 46a4b48..94a6932 100644
--- a/av1/encoder/pickrst.h
+++ b/av1/encoder/pickrst.h
@@ -65,7 +65,7 @@
* \param[in] sd Source frame buffer
* \param[in,out] cpi Top-level encoder structure
*
- * \return Nothing is returned. Instead, chosen restoration filter
+ * \remark Nothing is returned. Instead, chosen restoration filter
* types and parameters are stored per plane in the \c rst_info structure
* of type \ref RestorationInfo inside \c cpi->common:
* \arg \c rst_info[ \c 0 ]: Chosen parameters for Y plane
diff --git a/av1/encoder/ratectrl.c b/av1/encoder/ratectrl.c
index a7d309e..9e2bfc1 100644
--- a/av1/encoder/ratectrl.c
+++ b/av1/encoder/ratectrl.c
@@ -606,7 +606,7 @@
* \param[in] width Frame width
* \param[in] height Frame height
*
- * \return None but updates the rate correction factor for the
+ * \remark Updates the rate correction factor for the
* current frame type in cpi->rc.
*/
static void set_rate_correction_factor(AV1_COMP *cpi, int is_encode_stage,
@@ -658,6 +658,8 @@
double adjustment_limit;
const int MBs = av1_get_MBs(width, height);
int projected_size_based_on_q = 0;
+ int cyclic_refresh_active =
+ cpi->oxcf.q_cfg.aq_mode == CYCLIC_REFRESH_AQ && cpi->common.seg.enabled;
// Do not update the rate factors for arf overlay frames.
if (cpi->rc.is_src_frame_alt_ref) return;
@@ -667,7 +669,7 @@
// Work out how big we would have expected the frame to be at this Q given
// the current correction factor.
// Stay in double to avoid int overflow when values are large
- if (cpi->oxcf.q_cfg.aq_mode == CYCLIC_REFRESH_AQ && cpi->common.seg.enabled) {
+ if (cyclic_refresh_active) {
projected_size_based_on_q =
av1_cyclic_refresh_estimate_bits_at_q(cpi, rate_correction_factor);
} else {
@@ -682,7 +684,17 @@
(double)projected_size_based_on_q;
// Clamp correction factor to prevent anything too extreme
- correction_factor = AOMMIN(AOMMAX(correction_factor, 0.25), 4.0);
+ correction_factor = AOMMAX(correction_factor, 0.25);
+
+ cpi->rc.q_2_frame = cpi->rc.q_1_frame;
+ cpi->rc.q_1_frame = cm->quant_params.base_qindex;
+ cpi->rc.rc_2_frame = cpi->rc.rc_1_frame;
+ if (correction_factor > 1.1)
+ cpi->rc.rc_1_frame = -1;
+ else if (correction_factor < 0.9)
+ cpi->rc.rc_1_frame = 1;
+ else
+ cpi->rc.rc_1_frame = 0;
// Decide how heavily to dampen the adjustment
if (correction_factor > 0.0) {
@@ -697,15 +709,23 @@
adjustment_limit = 0.75;
}
- cpi->rc.q_2_frame = cpi->rc.q_1_frame;
- cpi->rc.q_1_frame = cm->quant_params.base_qindex;
- cpi->rc.rc_2_frame = cpi->rc.rc_1_frame;
- if (correction_factor > 1.1)
- cpi->rc.rc_1_frame = -1;
- else if (correction_factor < 0.9)
- cpi->rc.rc_1_frame = 1;
- else
- cpi->rc.rc_1_frame = 0;
+ // Adjustment to delta Q and number of blocks updated in cyclic refressh
+ // based on over or under shoot of target in current frame.
+ if (cyclic_refresh_active && (cpi->rc.this_frame_target > 0) &&
+ !cpi->ppi->use_svc) {
+ CYCLIC_REFRESH *const cr = cpi->cyclic_refresh;
+ if (correction_factor > 1.25) {
+ cr->percent_refresh_adjustment =
+ AOMMAX(cr->percent_refresh_adjustment - 1, -5);
+ cr->rate_ratio_qdelta_adjustment =
+ AOMMAX(cr->rate_ratio_qdelta_adjustment - 0.05, -0.0);
+ } else if (correction_factor < 0.5) {
+ cr->percent_refresh_adjustment =
+ AOMMIN(cr->percent_refresh_adjustment + 1, 5);
+ cr->rate_ratio_qdelta_adjustment =
+ AOMMIN(cr->rate_ratio_qdelta_adjustment + 0.05, 0.25);
+ }
+ }
if (correction_factor > 1.01) {
// We are not already at the worst allowable quality
@@ -1487,7 +1507,7 @@
double q_adj_factor = 1.0;
double q_val;
- // Baseline value derived from cpi->active_worst_quality and kf boost.
+ // Baseline value derived from active_worst_quality and kf boost.
active_best_quality =
get_kf_active_quality(p_rc, active_worst_quality, bit_depth);
if (cpi->is_screen_content_type) {
@@ -1935,8 +1955,8 @@
return q;
}
-int av1_rc_pick_q_and_bounds(const AV1_COMP *cpi, int width, int height,
- int gf_index, int *bottom_index, int *top_index) {
+int av1_rc_pick_q_and_bounds(AV1_COMP *cpi, int width, int height, int gf_index,
+ int *bottom_index, int *top_index) {
PRIMARY_RATE_CONTROL *const p_rc = &cpi->ppi->p_rc;
int q;
// TODO(sarahparker) merge no-stats vbr and altref q computation
@@ -1948,6 +1968,9 @@
if (cpi->oxcf.rc_cfg.mode == AOM_CBR) {
q = rc_pick_q_and_bounds_no_stats_cbr(cpi, width, height, bottom_index,
top_index);
+ // preserve copy of active worst quality selected.
+ cpi->rc.active_worst_quality = *top_index;
+
#if USE_UNRESTRICTED_Q_IN_CQ_MODE
} else if (cpi->oxcf.rc_cfg.mode == AOM_CQ) {
q = rc_pick_q_and_bounds_no_stats_cq(cpi, width, height, bottom_index,
@@ -2536,7 +2559,7 @@
void av1_adjust_gf_refresh_qp_one_pass_rt(AV1_COMP *cpi) {
AV1_COMMON *const cm = &cpi->common;
RATE_CONTROL *const rc = &cpi->rc;
- SVC *const svc = &cpi->svc;
+ RTC_REF *const rtc_ref = &cpi->rtc_ref;
const int resize_pending = is_frame_resize_pending(cpi);
if (!resize_pending && !rc->high_source_sad) {
// Check if we should disable GF refresh (if period is up),
@@ -2551,7 +2574,7 @@
if (rc->frames_till_gf_update_due == 1 &&
cm->quant_params.base_qindex > avg_qp) {
// Disable GF refresh since QP is above the runninhg average QP.
- svc->refresh[svc->gld_idx_1layer] = 0;
+ rtc_ref->refresh[rtc_ref->gld_idx_1layer] = 0;
gf_update_changed = 1;
cpi->refresh_frame.golden_frame = 0;
} else if (allow_gf_update &&
@@ -2559,7 +2582,7 @@
(rc->avg_frame_low_motion && rc->avg_frame_low_motion < 20))) {
// Force refresh since QP is well below average QP or this is a high
// motion frame.
- svc->refresh[svc->gld_idx_1layer] = 1;
+ rtc_ref->refresh[rtc_ref->gld_idx_1layer] = 1;
gf_update_changed = 1;
cpi->refresh_frame.golden_frame = 1;
}
@@ -2567,8 +2590,9 @@
set_baseline_gf_interval(cpi, INTER_FRAME);
int refresh_mask = 0;
for (unsigned int i = 0; i < INTER_REFS_PER_FRAME; i++) {
- int ref_frame_map_idx = svc->ref_idx[i];
- refresh_mask |= svc->refresh[ref_frame_map_idx] << ref_frame_map_idx;
+ int ref_frame_map_idx = rtc_ref->ref_idx[i];
+ refresh_mask |= rtc_ref->refresh[ref_frame_map_idx]
+ << ref_frame_map_idx;
}
cm->current_frame.refresh_frame_flags = refresh_mask;
}
@@ -2587,19 +2611,18 @@
* \param[in] cpi Top level encoder structure
* \param[in] gf_update Flag to indicate if GF is updated
*
- * \return Nothing is returned. Instead the settings for the prediction
+ * \remark Nothing is returned. Instead the settings for the prediction
* structure are set in \c cpi-ext_flags; and the buffer slot index
* (for each of 7 references) and refresh flags (for each of the 8 slots)
* are set in \c cpi->svc.ref_idx[] and \c cpi->svc.refresh[].
*/
-void av1_set_reference_structure_one_pass_rt(AV1_COMP *cpi, int gf_update) {
+void av1_set_rtc_reference_structure_one_layer(AV1_COMP *cpi, int gf_update) {
AV1_COMMON *const cm = &cpi->common;
ExternalFlags *const ext_flags = &cpi->ext_flags;
RATE_CONTROL *const rc = &cpi->rc;
ExtRefreshFrameFlagsInfo *const ext_refresh_frame_flags =
&ext_flags->refresh_frame;
- SVC *const svc = &cpi->svc;
- const int gld_fixed_slot = 1;
+ RTC_REF *const rtc_ref = &cpi->rtc_ref;
unsigned int lag_alt = 4;
int last_idx = 0;
int last_idx_refresh = 0;
@@ -2607,7 +2630,6 @@
int alt_ref_idx = 0;
int last2_idx = 0;
ext_refresh_frame_flags->update_pending = 1;
- svc->set_ref_frame_config = 1;
ext_flags->ref_frame_flags = 0;
ext_refresh_frame_flags->last_frame = 1;
ext_refresh_frame_flags->golden_frame = 0;
@@ -2630,33 +2652,28 @@
else if (rc->avg_source_sad > th_frame_sad[th_idx][2])
lag_alt = 5;
}
- for (int i = 0; i < INTER_REFS_PER_FRAME; ++i) svc->ref_idx[i] = 7;
- for (int i = 0; i < REF_FRAMES; ++i) svc->refresh[i] = 0;
+ // This defines the reference structure for 1 layer (non-svc) RTC encoding.
+ // To avoid the internal/default reference structure for non-realtime
+ // overwriting this behavior, we use the "svc" ref parameters from the
+ // external control SET_SVC_REF_FRAME_CONFIG.
+ // TODO(marpan): rename that control and the related internal parameters
+ // to rtc_ref.
+ for (int i = 0; i < INTER_REFS_PER_FRAME; ++i) rtc_ref->ref_idx[i] = 7;
+ for (int i = 0; i < REF_FRAMES; ++i) rtc_ref->refresh[i] = 0;
// Set the reference frame flags.
ext_flags->ref_frame_flags ^= AOM_LAST_FLAG;
ext_flags->ref_frame_flags ^= AOM_ALT_FLAG;
ext_flags->ref_frame_flags ^= AOM_GOLD_FLAG;
if (cpi->sf.rt_sf.ref_frame_comp_nonrd[1])
ext_flags->ref_frame_flags ^= AOM_LAST2_FLAG;
- const int sh = 7 - gld_fixed_slot;
+ const int sh = 6;
// Moving index slot for last: 0 - (sh - 1).
if (cm->current_frame.frame_number > 1)
last_idx = ((cm->current_frame.frame_number - 1) % sh);
// Moving index for refresh of last: one ahead for next frame.
last_idx_refresh = (cm->current_frame.frame_number % sh);
gld_idx = 6;
- if (!gld_fixed_slot) {
- gld_idx = 7;
- const unsigned int lag_gld = 7; // Must be <= 7.
- // Moving index for gld_ref, lag behind current by gld_interval frames.
- if (cm->current_frame.frame_number > lag_gld)
- gld_idx = ((cm->current_frame.frame_number - lag_gld) % sh);
- // When golden is not long-term reference with fixed slot update but
- // a reference with a moving slot with fixed lag behind last
- // (i.e., gld_fixed_slot = 0), we should disable the
- // gf_refresh_based_on_qp feature.
- cpi->sf.rt_sf.gf_refresh_based_on_qp = 0;
- }
+
// Moving index for alt_ref, lag behind LAST by lag_alt frames.
if (cm->current_frame.frame_number > lag_alt)
alt_ref_idx = ((cm->current_frame.frame_number - lag_alt) % sh);
@@ -2665,32 +2682,31 @@
if (cm->current_frame.frame_number > 2)
last2_idx = ((cm->current_frame.frame_number - 2) % sh);
}
- svc->ref_idx[0] = last_idx; // LAST
- svc->ref_idx[1] = last_idx_refresh; // LAST2 (for refresh of last).
+ rtc_ref->ref_idx[0] = last_idx; // LAST
+ rtc_ref->ref_idx[1] = last_idx_refresh; // LAST2 (for refresh of last).
if (cpi->sf.rt_sf.ref_frame_comp_nonrd[1]) {
- svc->ref_idx[1] = last2_idx; // LAST2
- svc->ref_idx[2] = last_idx_refresh; // LAST3 (for refresh of last).
+ rtc_ref->ref_idx[1] = last2_idx; // LAST2
+ rtc_ref->ref_idx[2] = last_idx_refresh; // LAST3 (for refresh of last).
}
- svc->ref_idx[3] = gld_idx; // GOLDEN
- svc->ref_idx[6] = alt_ref_idx; // ALT_REF
+ rtc_ref->ref_idx[3] = gld_idx; // GOLDEN
+ rtc_ref->ref_idx[6] = alt_ref_idx; // ALT_REF
// Refresh this slot, which will become LAST on next frame.
- svc->refresh[last_idx_refresh] = 1;
+ rtc_ref->refresh[last_idx_refresh] = 1;
// Update GOLDEN on period for fixed slot case.
- if (gld_fixed_slot && gf_update &&
- cm->current_frame.frame_type != KEY_FRAME) {
+ if (gf_update && cm->current_frame.frame_type != KEY_FRAME) {
ext_refresh_frame_flags->golden_frame = 1;
- svc->refresh[gld_idx] = 1;
+ rtc_ref->refresh[gld_idx] = 1;
}
- svc->gld_idx_1layer = gld_idx;
+ rtc_ref->gld_idx_1layer = gld_idx;
// Set the flag to reduce the number of reference frame buffers used.
// This assumes that slot 7 is never used.
cpi->rt_reduce_num_ref_buffers = 1;
- cpi->rt_reduce_num_ref_buffers &= (svc->ref_idx[0] < 7);
- cpi->rt_reduce_num_ref_buffers &= (svc->ref_idx[1] < 7);
- cpi->rt_reduce_num_ref_buffers &= (svc->ref_idx[3] < 7);
- cpi->rt_reduce_num_ref_buffers &= (svc->ref_idx[6] < 7);
+ cpi->rt_reduce_num_ref_buffers &= (rtc_ref->ref_idx[0] < 7);
+ cpi->rt_reduce_num_ref_buffers &= (rtc_ref->ref_idx[1] < 7);
+ cpi->rt_reduce_num_ref_buffers &= (rtc_ref->ref_idx[3] < 7);
+ cpi->rt_reduce_num_ref_buffers &= (rtc_ref->ref_idx[6] < 7);
if (cpi->sf.rt_sf.ref_frame_comp_nonrd[1])
- cpi->rt_reduce_num_ref_buffers &= (svc->ref_idx[2] < 7);
+ cpi->rt_reduce_num_ref_buffers &= (rtc_ref->ref_idx[2] < 7);
}
/*!\brief Check for scene detection, for 1 pass real-time mode.
@@ -2703,7 +2719,7 @@
* \param[in] cpi Top level encoder structure
* \param[in] frame_input Current and last input source frames
*
- * \return Nothing is returned. Instead the flag \c cpi->rc.high_source_sad
+ * \remark Nothing is returned. Instead the flag \c cpi->rc.high_source_sad
* is set if scene change is detected, and \c cpi->rc.avg_source_sad is updated.
*/
static void rc_scene_detection_onepass_rt(AV1_COMP *cpi,
@@ -2772,8 +2788,8 @@
// Flag to check light change or not.
const int check_light_change = 0;
// Store blkwise SAD for later use
- if (cpi->sf.rt_sf.sad_based_comp_prune && (cm->spatial_layer_id == 0) &&
- (cm->width == cm->render_width) && (cm->height == cm->render_height)) {
+ if ((cm->spatial_layer_id == 0) && (cm->width == cm->render_width) &&
+ (cm->height == cm->render_height)) {
full_sampling = 1;
if (cpi->src_sad_blk_64x64 == NULL) {
CHECK_MEM_ERROR(
@@ -2930,7 +2946,7 @@
* \ingroup rate_control
* \param[in] cpi Top level encoder structure
*
- * \return Return resized width/height in \c cpi->resize_pending_params,
+ * \remark Return resized width/height in \c cpi->resize_pending_params,
* and update some resize counters in \c rc.
*/
static void dynamic_resize_one_pass_cbr(AV1_COMP *cpi) {
@@ -3042,8 +3058,7 @@
return 0;
}
-void av1_get_one_pass_rt_params(AV1_COMP *cpi,
- EncodeFrameParams *const frame_params,
+void av1_get_one_pass_rt_params(AV1_COMP *cpi, FRAME_TYPE *const frame_type,
const EncodeFrameInput *frame_input,
unsigned int frame_flags) {
RATE_CONTROL *const rc = &cpi->rc;
@@ -3065,7 +3080,7 @@
}
// Set frame type.
if (set_key_frame(cpi, frame_flags)) {
- frame_params->frame_type = KEY_FRAME;
+ *frame_type = KEY_FRAME;
p_rc->this_key_frame_forced =
cm->current_frame.frame_number != 0 && rc->frames_to_key == 0;
rc->frames_to_key = cpi->oxcf.kf_cfg.key_freq_max;
@@ -3079,7 +3094,7 @@
svc->layer_context[layer].is_key_frame = 1;
}
} else {
- frame_params->frame_type = INTER_FRAME;
+ *frame_type = INTER_FRAME;
gf_group->update_type[cpi->gf_frame_index] = LF_UPDATE;
gf_group->frame_type[cpi->gf_frame_index] = INTER_FRAME;
gf_group->refbuf_state[cpi->gf_frame_index] = REFBUF_UPDATE;
@@ -3089,12 +3104,13 @@
svc->spatial_layer_id == 0
? 0
: svc->layer_context[svc->temporal_layer_id].is_key_frame;
- // If the user is setting the SVC pattern with set_ref_frame_config and
- // did not set any references, set the frame type to Intra-only.
- if (svc->set_ref_frame_config) {
+ // If the user is setting the reference structure with
+ // set_ref_frame_config and did not set any references, set the
+ // frame type to Intra-only.
+ if (cpi->rtc_ref.set_ref_frame_config) {
int no_references_set = 1;
for (int i = 0; i < INTER_REFS_PER_FRAME; i++) {
- if (svc->reference[i]) {
+ if (cpi->rtc_ref.reference[i]) {
no_references_set = 0;
break;
}
@@ -3103,7 +3119,7 @@
// The stream can start decoding on INTRA_ONLY_FRAME so long as the
// layer with the intra_only_frame doesn't signal a reference to a slot
// that hasn't been set yet.
- if (no_references_set) frame_params->frame_type = INTRA_ONLY_FRAME;
+ if (no_references_set) *frame_type = INTRA_ONLY_FRAME;
}
}
}
@@ -3132,19 +3148,17 @@
}
// Set the GF interval and update flag.
if (!rc->rtc_external_ratectrl)
- set_gf_interval_update_onepass_rt(cpi, frame_params->frame_type);
+ set_gf_interval_update_onepass_rt(cpi, *frame_type);
// Set target size.
if (cpi->oxcf.rc_cfg.mode == AOM_CBR) {
- if (frame_params->frame_type == KEY_FRAME ||
- frame_params->frame_type == INTRA_ONLY_FRAME) {
+ if (*frame_type == KEY_FRAME || *frame_type == INTRA_ONLY_FRAME) {
target = av1_calc_iframe_target_size_one_pass_cbr(cpi);
} else {
target = av1_calc_pframe_target_size_one_pass_cbr(
cpi, gf_group->update_type[cpi->gf_frame_index]);
}
} else {
- if (frame_params->frame_type == KEY_FRAME ||
- frame_params->frame_type == INTRA_ONLY_FRAME) {
+ if (*frame_type == KEY_FRAME || *frame_type == INTRA_ONLY_FRAME) {
target = av1_calc_iframe_target_size_one_pass_vbr(cpi);
} else {
target = av1_calc_pframe_target_size_one_pass_vbr(
@@ -3156,7 +3170,7 @@
av1_rc_set_frame_target(cpi, target, cm->width, cm->height);
rc->base_frame_target = target;
- cm->current_frame.frame_type = frame_params->frame_type;
+ cm->current_frame.frame_type = *frame_type;
// For fixed mode SVC: if KSVC is enabled remove inter layer
// prediction on spatial enhancement layer frames for frames
// whose base is not KEY frame.
diff --git a/av1/encoder/ratectrl.h b/av1/encoder/ratectrl.h
index 6dc5baf..6a678b7 100644
--- a/av1/encoder/ratectrl.h
+++ b/av1/encoder/ratectrl.h
@@ -228,7 +228,7 @@
/*!\endcond */
/*!
- * Proposed maximum alloed Q for current frame
+ * Proposed maximum allowed Q for current frame
*/
int active_worst_quality;
@@ -539,6 +539,8 @@
int q_history[MAX_Q_HISTORY];
} PRIMARY_RATE_CONTROL;
+/*!\cond */
+
struct AV1_COMP;
struct AV1EncoderConfig;
struct GF_GROUP;
@@ -582,7 +584,6 @@
// Functions to set parameters for encoding before the actual
// encode_frame_to_data_rate() function.
-struct EncodeFrameParams;
struct EncodeFrameInput;
// Post encode update of the rate control parameters based
@@ -603,7 +604,7 @@
* \param[in] width Frame width
* \param[in] height Frame height
*
- * \return None but updates the relevant rate correction factor in cpi->rc
+ * \remark Updates the relevant rate correction factor in cpi->rc
*/
void av1_rc_update_rate_correction_factors(struct AV1_COMP *cpi,
int is_encode_stage, int width,
@@ -634,7 +635,7 @@
* \return Returns selected q index to be used for encoding this frame.
* Also, updates \c rc->arf_q.
*/
-int av1_rc_pick_q_and_bounds(const struct AV1_COMP *cpi, int width, int height,
+int av1_rc_pick_q_and_bounds(struct AV1_COMP *cpi, int width, int height,
int gf_index, int *bottom_index, int *top_index);
/*!\brief Estimates q to achieve a target bits per frame
@@ -700,8 +701,8 @@
void av1_adjust_gf_refresh_qp_one_pass_rt(struct AV1_COMP *cpi);
-void av1_set_reference_structure_one_pass_rt(struct AV1_COMP *cpi,
- int gf_update);
+void av1_set_rtc_reference_structure_one_layer(struct AV1_COMP *cpi,
+ int gf_update);
/*!\endcond */
/*!\brief Calculates how many bits to use for a P frame in one pass vbr
@@ -766,15 +767,16 @@
*
* \ingroup rate_control
* \param[in] cpi Top level encoder structure
- * \param[in] frame_params Encoder frame parameters
+ * \param[in] frame_type Encoder frame type
* \param[in] frame_input Current and last input source frames
- * \param[in] frame_flags Emcoder frame flags
+ * \param[in] frame_flags Encoder frame flags
*
- * \return Nothing is returned. Instead the settings computed in this
- * funtion are set in: \c frame_params, \c cpi->common, \c cpi->rc, \c cpi->svc.
+ * \remark Nothing is returned. Instead the settings computed in this
+ * function are set in: \c frame_params, \c cpi->common, \c cpi->rc,
+ * \c cpi->svc.
*/
void av1_get_one_pass_rt_params(struct AV1_COMP *cpi,
- struct EncodeFrameParams *const frame_params,
+ FRAME_TYPE *const frame_type,
const struct EncodeFrameInput *frame_input,
unsigned int frame_flags);
diff --git a/av1/encoder/rd.c b/av1/encoder/rd.c
index 665ea48..98c4fbe 100644
--- a/av1/encoder/rd.c
+++ b/av1/encoder/rd.c
@@ -32,6 +32,7 @@
#include "av1/encoder/cost.h"
#include "av1/encoder/encodemv.h"
#include "av1/encoder/encoder.h"
+#include "av1/encoder/nonrd_opt.h"
#include "av1/encoder/ratectrl.h"
#include "av1/encoder/rd.h"
@@ -507,8 +508,26 @@
}
}
-static void set_block_thresholds(const AV1_COMMON *cm, RD_OPT *rd) {
+static void set_block_thresholds(const AV1_COMMON *cm, RD_OPT *rd,
+ int use_nonrd_pick_mode) {
int i, bsize, segment_id;
+ THR_MODES mode_indices[RTC_REFS * RTC_MODES] = { 0 };
+ int num_modes_count = use_nonrd_pick_mode ? 0 : MAX_MODES;
+
+ if (use_nonrd_pick_mode) {
+ for (int r_idx = 0; r_idx < RTC_REFS; r_idx++) {
+ const MV_REFERENCE_FRAME ref = real_time_ref_combos[r_idx][0];
+ if (ref != INTRA_FRAME) {
+ for (i = 0; i < RTC_INTER_MODES; i++)
+ mode_indices[num_modes_count++] =
+ mode_idx[ref][mode_offset(inter_mode_list[i])];
+ } else {
+ for (i = 0; i < RTC_INTRA_MODES; i++)
+ mode_indices[num_modes_count++] =
+ mode_idx[ref][mode_offset(intra_mode_list[i])];
+ }
+ }
+ }
for (segment_id = 0; segment_id < MAX_SEGMENTS; ++segment_id) {
const int qindex = clamp(
@@ -523,10 +542,13 @@
const int t = q * rd_thresh_block_size_factor[bsize];
const int thresh_max = INT_MAX / t;
- for (i = 0; i < MAX_MODES; ++i)
- rd->threshes[segment_id][bsize][i] = rd->thresh_mult[i] < thresh_max
- ? rd->thresh_mult[i] * t / 4
- : INT_MAX;
+ for (i = 0; i < num_modes_count; ++i) {
+ const int mode_index = use_nonrd_pick_mode ? mode_indices[i] : i;
+ rd->threshes[segment_id][bsize][mode_index] =
+ rd->thresh_mult[mode_index] < thresh_max
+ ? rd->thresh_mult[mode_index] * t / 4
+ : INT_MAX;
+ }
}
}
}
@@ -748,7 +770,7 @@
av1_set_error_per_bit(&x->errorperbit, rd->RDMULT);
- set_block_thresholds(cm, rd);
+ set_block_thresholds(cm, rd, cpi->sf.rt_sf.use_nonrd_pick_mode);
populate_unified_cost_update_freq(cpi->oxcf.cost_upd_freq, sf);
const INTER_MODE_SPEED_FEATURES *const inter_sf = &cpi->sf.inter_sf;
diff --git a/av1/encoder/rd.h b/av1/encoder/rd.h
index 8d0277e..96d53c9 100644
--- a/av1/encoder/rd.h
+++ b/av1/encoder/rd.h
@@ -56,6 +56,28 @@
// Factor to weigh the rate for switchable interp filters.
#define SWITCHABLE_INTERP_RATE_FACTOR 1
+#define RTC_REFS 4
+static const MV_REFERENCE_FRAME real_time_ref_combos[RTC_REFS][2] = {
+ { LAST_FRAME, NONE_FRAME },
+ { ALTREF_FRAME, NONE_FRAME },
+ { GOLDEN_FRAME, NONE_FRAME },
+ { INTRA_FRAME, NONE_FRAME }
+};
+
+static INLINE int mode_offset(const PREDICTION_MODE mode) {
+ if (mode >= NEARESTMV) {
+ return INTER_OFFSET(mode);
+ } else {
+ switch (mode) {
+ case DC_PRED: return 0;
+ case V_PRED: return 1;
+ case H_PRED: return 2;
+ case SMOOTH_PRED: return 3;
+ default: assert(0); return -1;
+ }
+ }
+}
+
enum {
// Default initialization when we are not using winner mode framework. e.g.
// intrabc
diff --git a/av1/encoder/rdopt.c b/av1/encoder/rdopt.c
index a51b200..3085e2d 100644
--- a/av1/encoder/rdopt.c
+++ b/av1/encoder/rdopt.c
@@ -3295,7 +3295,7 @@
const int num_planes = av1_num_planes(cm);
TxfmSearchInfo *txfm_info = &x->txfm_search_info;
int rate_y = 0, rate_uv = 0, rate_y_tokenonly = 0, rate_uv_tokenonly = 0;
- int y_skip_txfm = 0, uv_skip_txfm = 0;
+ uint8_t y_skip_txfm = 0, uv_skip_txfm = 0;
int64_t dist_y = 0, dist_uv = 0;
ctx->rd_stats.skip_txfm = 0;
@@ -3703,13 +3703,6 @@
{ ALTREF_FRAME, INTRA_FRAME }, { BWDREF_FRAME, INTRA_FRAME },
};
-static const MV_REFERENCE_FRAME real_time_ref_combos[][2] = {
- { LAST_FRAME, NONE_FRAME },
- { ALTREF_FRAME, NONE_FRAME },
- { GOLDEN_FRAME, NONE_FRAME },
- { INTRA_FRAME, NONE_FRAME }
-};
-
typedef enum { REF_SET_FULL, REF_SET_REDUCED, REF_SET_REALTIME } REF_SET;
static AOM_INLINE void default_skip_mask(mode_skip_mask_t *mask,
@@ -3892,7 +3885,7 @@
}
mask->pred_modes[INTRA_FRAME] |=
- ~(sf->intra_sf.intra_y_mode_mask[max_txsize_lookup[bsize]]);
+ ~(uint32_t)sf->intra_sf.intra_y_mode_mask[max_txsize_lookup[bsize]];
}
static AOM_INLINE void init_neighbor_pred_buf(
@@ -5326,19 +5319,11 @@
* InterModeSearchState::intra_search_state so it can be reused later by \ref
* av1_search_palette_mode.
*
- * \return Returns the rdcost of the current intra-mode if it's available,
- * otherwise returns INT64_MAX. The corresponding values in x->e_mbd.mi[0],
- * rd_stats, rd_stats_y/uv, and best_intra_rd are also updated. Moreover, in the
- * first evocation of the function, the chroma intra mode result is cached in
- * intra_search_state to be used in subsequent calls. In the first evaluation
- * with directional mode, a prune_mask computed with histogram of gradient is
- * also stored in intra_search_state.
- *
* \param[in,out] search_state Struct keep track of the prediction mode
* search state in interframe.
*
* \param[in] cpi Top-level encoder structure.
- * \param[in] x Pointer to struct holding all the data for
+ * \param[in,out] x Pointer to struct holding all the data for
* the current prediction block.
* \param[out] rd_cost Stores the best rd_cost among all the
* prediction modes searched.
@@ -5346,21 +5331,21 @@
* \param[in,out] ctx Structure to hold the number of 4x4 blks to
* copy the tx_type and txfm_skip arrays.
* for only the Y plane.
- * \param[in,out] sf_args Stores the list of intra mode candidates
+ * \param[in] sf_args Stores the list of intra mode candidates
* to be searched.
* \param[in] intra_ref_frame_cost The entropy cost for signaling that the
* current ref frame is an intra frame.
* \param[in] yrd_threshold The rdcost threshold for luma intra mode to
* terminate chroma intra mode search.
*
- * \return Returns INT64_MAX if the determined motion mode is invalid and the
- * current motion mode being tested should be skipped. It returns 0 if the
- * motion mode search is a success.
+ * \remark If a new best mode is found, search_state and rd_costs are updated
+ * correspondingly. While x is also modified, it is only used as a temporary
+ * buffer, and the final decisions are stored in search_state.
*/
static AOM_INLINE void search_intra_modes_in_interframe(
InterModeSearchState *search_state, const AV1_COMP *cpi, MACROBLOCK *x,
RD_STATS *rd_cost, BLOCK_SIZE bsize, PICK_MODE_CONTEXT *ctx,
- InterModeSFArgs *sf_args, unsigned int intra_ref_frame_cost,
+ const InterModeSFArgs *sf_args, unsigned int intra_ref_frame_cost,
int64_t yrd_threshold) {
const AV1_COMMON *const cm = &cpi->common;
const SPEED_FEATURES *const sf = &cpi->sf;
diff --git a/av1/encoder/rdopt.h b/av1/encoder/rdopt.h
index 2fead8f..4d813a2 100644
--- a/av1/encoder/rdopt.h
+++ b/av1/encoder/rdopt.h
@@ -54,7 +54,7 @@
during the mode picking process.
* \param[in] best_rd Best RD seen for this block so far.
*
- * \return Nothing is returned. Instead, the MB_MODE_INFO struct inside x
+ * \remark Nothing is returned. Instead, the MB_MODE_INFO struct inside x
* is modified to store information about the best mode computed
* in this function. The rd_cost struct is also updated with the RD stats
* corresponding to the best mode found.
@@ -85,7 +85,7 @@
during the mode picking process
* \param[in] best_rd_so_far Best RD seen for this block so far
*
- * \return Nothing is returned. Instead, the MB_MODE_INFO struct inside x
+ * \remark Nothing is returned. Instead, the MB_MODE_INFO struct inside x
* is modified to store information about the best mode computed
* in this function. The rd_cost struct is also updated with the RD stats
* corresponding to the best mode found.
@@ -115,7 +115,7 @@
* \param[in] ctx Structure to hold snapshot of coding context
during the mode picking process
*
- * \return Nothing is returned. Instead, the MB_MODE_INFO struct inside x
+ * \remark Nothing is returned. Instead, the MB_MODE_INFO struct inside x
* is modified to store information about the best mode computed
* in this function. The rd_cost struct is also updated with the RD stats
* corresponding to the best mode found.
@@ -147,7 +147,7 @@
* \param[in] ctx Structure to hold snapshot of coding context
during the mode picking process
*
- * \return Nothing is returned. Instead, the MB_MODE_INFO struct inside x
+ * \remark Nothing is returned. Instead, the MB_MODE_INFO struct inside x
* is modified to store information about the best mode computed
* in this function. The rd_cost struct is also updated with the RD stats
* corresponding to the best mode found.
diff --git a/av1/encoder/segmentation.c b/av1/encoder/segmentation.c
index d315838..4b4e787 100644
--- a/av1/encoder/segmentation.c
+++ b/av1/encoder/segmentation.c
@@ -35,7 +35,7 @@
void av1_disable_segfeature(struct segmentation *seg, int segment_id,
SEG_LVL_FEATURES feature_id) {
- seg->feature_mask[segment_id] &= ~(1 << feature_id);
+ seg->feature_mask[segment_id] &= ~(1u << feature_id);
}
void av1_clear_segdata(struct segmentation *seg, int segment_id,
diff --git a/av1/encoder/speed_features.c b/av1/encoder/speed_features.c
index b43f80f..7ffcad6 100644
--- a/av1/encoder/speed_features.c
+++ b/av1/encoder/speed_features.c
@@ -554,6 +554,7 @@
sf->rt_sf.hybrid_intra_pickmode = 0;
sf->rt_sf.var_part_split_threshold_shift = 9;
sf->rt_sf.vbp_prune_16x16_split_using_min_max_sub_blk_var = true;
+ sf->rt_sf.prune_h_pred_using_best_mode_so_far = true;
}
// As the speed feature prune_chroma_modes_using_luma_winner already
@@ -806,6 +807,7 @@
if (!is_480p_or_larger) {
sf->tx_sf.tx_type_search.fast_inter_tx_type_prob_thresh =
boosted ? INT_MAX : 250;
+ sf->part_sf.partition_search_breakout_dist_thr = (1 << 26);
}
if (is_480p_or_lesser) {
@@ -840,8 +842,10 @@
if (is_720p_or_larger) {
sf->part_sf.use_square_partition_only_threshold = BLOCK_32X32;
+ sf->part_sf.partition_search_breakout_dist_thr = (1 << 28);
} else {
sf->part_sf.use_square_partition_only_threshold = BLOCK_16X16;
+ sf->part_sf.partition_search_breakout_dist_thr = (1 << 26);
}
if (is_720p_or_larger) {
@@ -1102,10 +1106,9 @@
sf->winner_mode_sf.enable_winner_mode_for_coeff_opt = 1;
sf->winner_mode_sf.enable_winner_mode_for_use_tx_domain_dist = 1;
sf->winner_mode_sf.motion_mode_for_winner_cand =
- boosted ? 0
- : gf_group->update_type[cpi->gf_frame_index] == INTNL_ARF_UPDATE
- ? 1
- : 2;
+ boosted ? 0
+ : gf_group->update_type[cpi->gf_frame_index] == INTNL_ARF_UPDATE ? 1
+ : 2;
sf->winner_mode_sf.prune_winner_mode_eval_level = boosted ? 0 : 4;
// For screen content, "prune_sgr_based_on_wiener = 2" cause large quality
@@ -1138,7 +1141,6 @@
sf->inter_sf.prune_ext_comp_using_neighbors = 2;
sf->inter_sf.prune_obmc_prob_thresh = INT_MAX;
sf->inter_sf.disable_interinter_wedge_var_thresh = UINT_MAX;
- sf->inter_sf.prune_nearest_near_mv_using_refmv_weight = boosted ? 0 : 1;
sf->interp_sf.cb_pred_filter_search = 1;
sf->interp_sf.skip_sharp_interp_filter_search = 1;
@@ -1227,8 +1229,9 @@
sf->part_sf.prune_rectangular_split_based_on_qidx =
boosted || allow_screen_content_tools ? 0 : 2;
sf->part_sf.prune_sub_8x8_partition_level =
- allow_screen_content_tools ? 0
- : frame_is_intra_only(&cpi->common) ? 1 : 2;
+ allow_screen_content_tools ? 0
+ : frame_is_intra_only(&cpi->common) ? 1
+ : 2;
sf->part_sf.prune_part4_search = 3;
sf->mv_sf.simple_motion_subpel_force_stop = FULL_PEL;
@@ -1261,11 +1264,13 @@
sf->rt_sf.prune_inter_modes_wrt_gf_arf_based_on_sad = 1;
if (speed >= 6)
sf->winner_mode_sf.prune_winner_mode_eval_level = boosted ? 0 : 2;
+ if (speed == 7) sf->rt_sf.prefer_large_partition_blocks = 2;
if (speed >= 7) {
sf->lpf_sf.cdef_pick_method = CDEF_PICK_FROM_Q;
sf->rt_sf.check_only_zero_zeromv_on_large_blocks = true;
+ sf->rt_sf.use_rtc_tf = 2;
}
- if (speed == 8) sf->rt_sf.prefer_large_partition_blocks = 2;
+ if (speed == 8) sf->rt_sf.prefer_large_partition_blocks = 1;
if (speed >= 8) {
sf->rt_sf.use_nonrd_filter_search = 0;
sf->rt_sf.tx_size_level_based_on_qstep = 1;
@@ -1274,6 +1279,14 @@
sf->rt_sf.use_comp_ref_nonrd = 0;
sf->rt_sf.nonrd_agressive_skip = 1;
sf->rt_sf.skip_intra_pred = 1;
+ // Only turn on enable_ref_short_signaling for low resolution when only
+ // LAST and GOLDEN ref frames are used.
+ sf->rt_sf.enable_ref_short_signaling =
+ (!sf->rt_sf.use_nonrd_altref_frame &&
+ (!sf->rt_sf.use_comp_ref_nonrd ||
+ (!sf->rt_sf.ref_frame_comp_nonrd[1] &&
+ !sf->rt_sf.ref_frame_comp_nonrd[2])));
+
// TODO(kyslov) Re-enable when AV1 models are trained
#if 0
#if CONFIG_RT_ML_PARTITIONING
@@ -1287,6 +1300,8 @@
if (speed >= 10) {
sf->rt_sf.skip_intra_pred = 2;
sf->rt_sf.hybrid_intra_pickmode = 3;
+ sf->rt_sf.reduce_mv_pel_precision_lowcomplex = 1;
+ sf->rt_sf.reduce_mv_pel_precision_highmotion = 1;
}
} else {
sf->rt_sf.prune_intra_mode_based_on_mv_range = 2;
@@ -1298,24 +1313,26 @@
}
if (speed == 6) sf->part_sf.disable_8x8_part_based_on_qidx = 1;
if (speed >= 6) sf->rt_sf.skip_newmv_mode_based_on_sse = 2;
+ if (speed == 7) sf->rt_sf.prefer_large_partition_blocks = 1;
+ if (speed >= 7) {
+ sf->rt_sf.use_rtc_tf = 1;
+ }
if (speed == 8 && !cpi->ppi->use_svc) {
sf->rt_sf.short_circuit_low_temp_var = 0;
sf->rt_sf.use_nonrd_altref_frame = 1;
}
- if (speed == 8) sf->rt_sf.prefer_large_partition_blocks = 3;
if (speed >= 8) sf->rt_sf.tx_size_level_based_on_qstep = 2;
if (speed >= 9) {
sf->rt_sf.gf_length_lvl = 1;
sf->rt_sf.skip_cdef_sb = 1;
sf->rt_sf.sad_based_adp_altref_lag = 2;
+ sf->rt_sf.reduce_mv_pel_precision_highmotion = 1;
}
-
if (speed >= 10) {
- // TODO(yunqing): extend this sf to other speeds and/or other resolutions.
- sf->rt_sf.use_rtc_tf = 1;
sf->rt_sf.hybrid_intra_pickmode = 2;
sf->rt_sf.sad_based_adp_altref_lag = 4;
sf->rt_sf.tx_size_level_based_on_qstep = 0;
+ sf->rt_sf.reduce_mv_pel_precision_highmotion = 2;
}
}
if (!is_480p_or_larger) {
@@ -1323,11 +1340,9 @@
sf->rt_sf.nonrd_check_partition_merge_mode = 2;
}
if (speed >= 8) {
- sf->mv_sf.subpel_search_method = SUBPEL_TREE;
sf->rt_sf.estimate_motion_for_var_based_partition = 1;
}
if (speed >= 9) {
- sf->mv_sf.subpel_search_method = SUBPEL_TREE_PRUNED;
sf->rt_sf.estimate_motion_for_var_based_partition = 0;
}
}
@@ -1337,17 +1352,18 @@
}
} else {
if (speed >= 6) sf->rt_sf.skip_newmv_mode_based_on_sse = 3;
- if (speed == 8) sf->rt_sf.prefer_large_partition_blocks = 1;
+ if (speed == 7) sf->rt_sf.prefer_large_partition_blocks = 0;
+ if (speed >= 7) sf->rt_sf.reduce_mv_pel_precision_lowcomplex = 2;
if (speed >= 9) {
sf->rt_sf.sad_based_adp_altref_lag = 1;
- sf->rt_sf.sad_based_comp_prune = 1;
+ sf->rt_sf.reduce_mv_pel_precision_lowcomplex = 0;
}
- if (speed >= 10) {
- sf->rt_sf.sad_based_adp_altref_lag = 3;
- sf->rt_sf.sad_based_comp_prune = 2;
- }
+ if (speed >= 10) sf->rt_sf.sad_based_adp_altref_lag = 3;
}
- if (cpi->ppi->use_svc) {
+ // Setting for SVC, or when the ref_frame_config control is
+ // used to set the reference structure.
+ if (cpi->ppi->use_svc || cpi->rtc_ref.set_ref_frame_config) {
+ const RTC_REF *const rtc_ref = &cpi->rtc_ref;
// For SVC: for greater than 2 temporal layers, use better mv search on
// base temporal layers, and only on base spatial layer if highest
// resolution is above 640x360.
@@ -1359,47 +1375,62 @@
sf->mv_sf.search_method = NSTEP;
sf->mv_sf.subpel_search_method = SUBPEL_TREE;
sf->rt_sf.fullpel_search_step_param = 6;
+ sf->rt_sf.reduce_mv_pel_precision_highmotion = 0;
}
- if (speed >= 9) {
+ if (speed >= 8) {
sf->rt_sf.disable_cdf_update_non_reference_frame = true;
- if (cpi->svc.non_reference_frame) sf->rt_sf.nonrd_agressive_skip = 1;
- }
- if (cpi->svc.ref_frame_comp[0] || cpi->svc.ref_frame_comp[1] ||
- cpi->svc.ref_frame_comp[2]) {
- sf->rt_sf.use_comp_ref_nonrd = 1;
- sf->rt_sf.ref_frame_comp_nonrd[0] =
- cpi->svc.ref_frame_comp[0] && cpi->svc.reference[GOLDEN_FRAME - 1];
- sf->rt_sf.ref_frame_comp_nonrd[1] =
- cpi->svc.ref_frame_comp[1] && cpi->svc.reference[LAST2_FRAME - 1];
- sf->rt_sf.ref_frame_comp_nonrd[2] =
- cpi->svc.ref_frame_comp[2] && cpi->svc.reference[ALTREF_FRAME - 1];
- } else {
- sf->rt_sf.use_comp_ref_nonrd = 0;
- sf->rt_sf.sad_based_comp_prune = 0;
+ sf->rt_sf.reduce_mv_pel_precision_highmotion = 2;
+ if (rtc_ref->non_reference_frame) {
+ sf->rt_sf.nonrd_agressive_skip = 1;
+ sf->mv_sf.subpel_search_method = SUBPEL_TREE_PRUNED_MORE;
+ }
}
if (speed <= 9 && cpi->svc.number_temporal_layers > 2 &&
cpi->svc.temporal_layer_id == 0)
sf->rt_sf.check_only_zero_zeromv_on_large_blocks = false;
else
sf->rt_sf.check_only_zero_zeromv_on_large_blocks = true;
+ if (cpi->svc.number_temporal_layers > 1 && cpi->svc.temporal_layer_id == 0)
+ sf->rt_sf.source_metrics_sb_nonrd = 0;
+ // Compound mode enabling.
+ if (rtc_ref->ref_frame_comp[0] || rtc_ref->ref_frame_comp[1] ||
+ rtc_ref->ref_frame_comp[2]) {
+ sf->rt_sf.use_comp_ref_nonrd = 1;
+ sf->rt_sf.ref_frame_comp_nonrd[0] =
+ rtc_ref->ref_frame_comp[0] && rtc_ref->reference[GOLDEN_FRAME - 1];
+ sf->rt_sf.ref_frame_comp_nonrd[1] =
+ rtc_ref->ref_frame_comp[1] && rtc_ref->reference[LAST2_FRAME - 1];
+ sf->rt_sf.ref_frame_comp_nonrd[2] =
+ rtc_ref->ref_frame_comp[2] && rtc_ref->reference[ALTREF_FRAME - 1];
+ } else {
+ sf->rt_sf.use_comp_ref_nonrd = 0;
+ }
}
+ // Screen settings.
if (cpi->oxcf.tune_cfg.content == AOM_CONTENT_SCREEN) {
// TODO(marpan): Check settings for speed 7 and 8.
if (speed >= 9) {
sf->rt_sf.prune_idtx_nonrd = 1;
- sf->rt_sf.part_early_exit_zeromv = 1;
+ sf->rt_sf.part_early_exit_zeromv = 2;
sf->rt_sf.skip_lf_screen = 1;
sf->rt_sf.use_nonrd_filter_search = 0;
sf->rt_sf.nonrd_prune_ref_frame_search = 3;
sf->rt_sf.var_part_split_threshold_shift = 10;
sf->mv_sf.subpel_search_method = SUBPEL_TREE_PRUNED_MORE;
- sf->rt_sf.force_half_pel_block = 1;
- sf->rt_sf.reduce_zeromv_mvres = true;
+ sf->rt_sf.reduce_mv_pel_precision_highmotion = 2;
+ sf->rt_sf.reduce_mv_pel_precision_lowcomplex = 1;
}
+ if (speed >= 10) {
+ if (cm->width * cm->height > 1920 * 1080)
+ sf->part_sf.disable_8x8_part_based_on_qidx = 1;
+ sf->rt_sf.set_zeromv_skip_based_on_source_sad = 2;
+ sf->rt_sf.screen_content_cdef_filter_qindex_thresh = 80;
+ sf->rt_sf.part_early_exit_zeromv = 1;
+ }
+ sf->rt_sf.use_nonrd_altref_frame = 0;
sf->rt_sf.skip_cdef_sb = 1;
sf->rt_sf.use_rtc_tf = 0;
sf->rt_sf.use_comp_ref_nonrd = 0;
- sf->rt_sf.sad_based_comp_prune = 0;
sf->rt_sf.source_metrics_sb_nonrd = 1;
if (cpi->rc.high_source_sad == 1) {
sf->rt_sf.prefer_large_partition_blocks = 0;
@@ -1417,8 +1448,6 @@
}
sf->rt_sf.partition_direct_merging = 0;
}
- if (cpi->svc.number_temporal_layers > 1 && cpi->svc.temporal_layer_id == 0)
- sf->rt_sf.source_metrics_sb_nonrd = 0;
}
// TODO(kyslov): now this is very similar to
@@ -1615,7 +1644,6 @@
sf->mv_sf.search_method = FAST_DIAMOND;
sf->mv_sf.subpel_force_stop = QUARTER_PEL;
- sf->mv_sf.subpel_search_method = SUBPEL_TREE_PRUNED;
sf->inter_sf.inter_mode_rd_model_estimation = 2;
// This sf is not applicable in non-rd path.
@@ -1667,7 +1695,10 @@
sf->winner_mode_sf.dc_blk_pred_level = 0;
sf->rt_sf.var_part_based_on_qidx = 3;
- sf->rt_sf.prune_global_globalmv_with_zeromv = true;
+ sf->rt_sf.prune_compoundmode_with_singlecompound_var = true;
+ sf->rt_sf.prune_compoundmode_with_singlemode_var = true;
+ sf->rt_sf.skip_compound_based_on_var = true;
+ sf->rt_sf.use_adaptive_subpel_search = true;
}
if (speed >= 8) {
@@ -1682,13 +1713,13 @@
sf->interp_sf.cb_pred_filter_search = 1;
sf->rt_sf.var_part_based_on_qidx = 4;
sf->rt_sf.partition_direct_merging = 1;
+ sf->rt_sf.prune_compoundmode_with_singlemode_var = false;
}
if (speed >= 9) {
sf->lpf_sf.cdef_pick_method = CDEF_PICK_FROM_Q;
sf->rt_sf.sse_early_term_inter_search = EARLY_TERM_IDX_3;
- sf->rt_sf.screen_content_cdef_filter_qindex_thresh = 20;
sf->rt_sf.estimate_motion_for_var_based_partition = 0;
- sf->rt_sf.prefer_large_partition_blocks = 4;
+ sf->rt_sf.prefer_large_partition_blocks = 3;
sf->rt_sf.skip_intra_pred = 2;
sf->rt_sf.var_part_split_threshold_shift = 9;
for (int i = 0; i < BLOCK_SIZES; ++i)
@@ -1696,6 +1727,15 @@
sf->rt_sf.var_part_based_on_qidx = 0;
sf->rt_sf.frame_level_mode_cost_update = true;
sf->rt_sf.check_only_zero_zeromv_on_large_blocks = true;
+ sf->rt_sf.reduce_mv_pel_precision_highmotion = 0;
+ sf->rt_sf.use_adaptive_subpel_search = false;
+ // For multi-thread use case with row_mt enabled, enable top right
+ // dependency wait of threads at mi level.
+ if ((cpi->oxcf.row_mt == 1) && (cpi->mt_info.num_workers > 1)) {
+ // TODO(Deepa): Disabled this feature due to enc/dec mismatch
+ // seen on the RTC 1080P set. Need to re-enable it after the fix.
+ sf->rt_sf.top_right_sync_wait_in_mis = false;
+ }
}
if (speed >= 10) {
sf->rt_sf.sse_early_term_inter_search = EARLY_TERM_IDX_4;
@@ -1703,9 +1743,6 @@
sf->rt_sf.nonrd_prune_ref_frame_search = 3;
sf->rt_sf.var_part_split_threshold_shift = 10;
sf->mv_sf.subpel_search_method = SUBPEL_TREE_PRUNED_MORE;
- sf->rt_sf.force_half_pel_block = 1;
- sf->rt_sf.reduce_zeromv_mvres = true;
- sf->rt_sf.screen_content_cdef_filter_qindex_thresh = 80;
}
}
@@ -1994,7 +2031,6 @@
rt_sf->use_comp_ref_nonrd = 0;
rt_sf->use_real_time_ref_set = 0;
rt_sf->short_circuit_low_temp_var = 0;
- rt_sf->use_modeled_non_rd_cost = 0;
rt_sf->reuse_inter_pred_nonrd = 0;
rt_sf->num_inter_modes_for_tx_search = INT_MAX;
rt_sf->use_nonrd_filter_search = 0;
@@ -2018,7 +2054,8 @@
rt_sf->prune_inter_modes_with_golden_ref = 0;
rt_sf->prune_inter_modes_wrt_gf_arf_based_on_sad = 0;
rt_sf->prune_inter_modes_using_temp_var = 0;
- rt_sf->force_half_pel_block = 0;
+ rt_sf->reduce_mv_pel_precision_highmotion = 0;
+ rt_sf->reduce_mv_pel_precision_lowcomplex = 0;
rt_sf->prune_intra_mode_based_on_mv_range = 0;
rt_sf->var_part_split_threshold_shift = 7;
rt_sf->gf_refresh_based_on_qp = 0;
@@ -2030,14 +2067,45 @@
rt_sf->sad_based_adp_altref_lag = 0;
rt_sf->partition_direct_merging = 0;
rt_sf->var_part_based_on_qidx = 0;
- rt_sf->sad_based_comp_prune = 0;
rt_sf->tx_size_level_based_on_qstep = 0;
- rt_sf->reduce_zeromv_mvres = false;
rt_sf->vbp_prune_16x16_split_using_min_max_sub_blk_var = false;
- rt_sf->prune_global_globalmv_with_zeromv = false;
+ rt_sf->prune_compoundmode_with_singlecompound_var = false;
rt_sf->frame_level_mode_cost_update = false;
+ rt_sf->prune_h_pred_using_best_mode_so_far = false;
rt_sf->check_only_zero_zeromv_on_large_blocks = false;
rt_sf->disable_cdf_update_non_reference_frame = false;
+ rt_sf->prune_compoundmode_with_singlemode_var = false;
+ rt_sf->skip_compound_based_on_var = false;
+ rt_sf->top_right_sync_wait_in_mis = false;
+ rt_sf->set_zeromv_skip_based_on_source_sad = 1;
+ rt_sf->use_adaptive_subpel_search = false;
+ rt_sf->screen_content_cdef_filter_qindex_thresh = 0;
+ rt_sf->enable_ref_short_signaling = false;
+}
+
+// Populate appropriate sub-pel search method based on speed feature and user
+// specified settings
+static void set_subpel_search_method(
+ MotionVectorSearchParams *mv_search_params,
+ unsigned int motion_vector_unit_test,
+ SUBPEL_SEARCH_METHODS subpel_search_method) {
+ if (subpel_search_method == SUBPEL_TREE) {
+ mv_search_params->find_fractional_mv_step = av1_find_best_sub_pixel_tree;
+ } else if (subpel_search_method == SUBPEL_TREE_PRUNED) {
+ mv_search_params->find_fractional_mv_step =
+ av1_find_best_sub_pixel_tree_pruned;
+ } else if (subpel_search_method == SUBPEL_TREE_PRUNED_MORE) {
+ mv_search_params->find_fractional_mv_step =
+ av1_find_best_sub_pixel_tree_pruned_more;
+ } else {
+ assert(0);
+ }
+
+ // This is only used in motion vector unit test.
+ if (motion_vector_unit_test == 1)
+ mv_search_params->find_fractional_mv_step = av1_return_max_sub_pixel_mv;
+ else if (motion_vector_unit_test == 2)
+ mv_search_params->find_fractional_mv_step = av1_return_min_sub_pixel_mv;
}
void av1_set_speed_features_framesize_dependent(AV1_COMP *cpi, int speed) {
@@ -2063,11 +2131,9 @@
(sf->inter_sf.disable_interintra_wedge_var_thresh != UINT_MAX);
}
- // This is only used in motion vector unit test.
- if (cpi->oxcf.unit_test_cfg.motion_vector_unit_test == 1)
- cpi->mv_search_params.find_fractional_mv_step = av1_return_max_sub_pixel_mv;
- else if (cpi->oxcf.unit_test_cfg.motion_vector_unit_test == 2)
- cpi->mv_search_params.find_fractional_mv_step = av1_return_min_sub_pixel_mv;
+ set_subpel_search_method(&cpi->mv_search_params,
+ cpi->oxcf.unit_test_cfg.motion_vector_unit_test,
+ sf->mv_sf.subpel_search_method);
// For multi-thread use case with row_mt enabled, cost update for a set of
// SB rows is not desirable. Hence, the sf mv_cost_upd_level is set to
@@ -2132,13 +2198,6 @@
(sf->inter_sf.disable_interintra_wedge_var_thresh != UINT_MAX);
}
- // sf->part_sf.partition_search_breakout_dist_thr is set assuming max 64x64
- // blocks. Normalise this if the blocks are bigger.
- if (MAX_SB_SIZE_LOG2 > 6) {
- sf->part_sf.partition_search_breakout_dist_thr <<=
- 2 * (MAX_SB_SIZE_LOG2 - 6);
- }
-
const int mesh_speed = AOMMIN(speed, MAX_MESH_SPEED);
for (i = 0; i < MAX_MESH_STEP; ++i) {
sf->mv_sf.mesh_patterns[i].range =
@@ -2166,22 +2225,9 @@
if (oxcf->pass == AOM_RC_ONE_PASS && has_no_stats_stage(cpi))
sf->hl_sf.recode_loop = DISALLOW_RECODE;
- MotionVectorSearchParams *const mv_search_params = &cpi->mv_search_params;
- if (sf->mv_sf.subpel_search_method == SUBPEL_TREE) {
- mv_search_params->find_fractional_mv_step = av1_find_best_sub_pixel_tree;
- } else if (sf->mv_sf.subpel_search_method == SUBPEL_TREE_PRUNED) {
- mv_search_params->find_fractional_mv_step =
- av1_find_best_sub_pixel_tree_pruned;
- } else if (sf->mv_sf.subpel_search_method == SUBPEL_TREE_PRUNED_MORE) {
- mv_search_params->find_fractional_mv_step =
- av1_find_best_sub_pixel_tree_pruned_more;
- }
-
- // This is only used in motion vector unit test.
- if (cpi->oxcf.unit_test_cfg.motion_vector_unit_test == 1)
- mv_search_params->find_fractional_mv_step = av1_return_max_sub_pixel_mv;
- else if (cpi->oxcf.unit_test_cfg.motion_vector_unit_test == 2)
- mv_search_params->find_fractional_mv_step = av1_return_min_sub_pixel_mv;
+ set_subpel_search_method(&cpi->mv_search_params,
+ cpi->oxcf.unit_test_cfg.motion_vector_unit_test,
+ sf->mv_sf.subpel_search_method);
// assert ensures that tx_domain_dist_level is accessed correctly
assert(cpi->sf.rd_sf.tx_domain_dist_thres_level >= 0 &&
@@ -2377,4 +2423,8 @@
sf->inter_sf.reuse_mask_search_results = 1;
}
}
+
+ set_subpel_search_method(&cpi->mv_search_params,
+ cpi->oxcf.unit_test_cfg.motion_vector_unit_test,
+ sf->mv_sf.subpel_search_method);
}
diff --git a/av1/encoder/speed_features.h b/av1/encoder/speed_features.h
index c55202d..60efc14 100644
--- a/av1/encoder/speed_features.h
+++ b/av1/encoder/speed_features.h
@@ -920,6 +920,14 @@
int prune_comp_using_best_single_mode_ref;
// Skip NEARESTMV and NEARMV using weight computed in ref mv list population
+ // This speed feature sometimes leads to severe visual artifacts for
+ // the overlay frame. It makes inter RD mode search skip NEARESTMV
+ // and NEARMV, and no valid inter mode is evaluated when the NEWMV mode
+ // is also early terminated due to the constraint that it does not handle
+ // zero mv difference. In this cases, intra modes will be chosen, leading
+ // to bad prediction and flickering artifacts.
+ // Turn off this feature for now. Be careful to check visual quality if
+ // anyone is going to turn it on.
int prune_nearest_near_mv_using_refmv_weight;
// Based on previous ref_mv_idx search result, prune the following search.
@@ -1423,9 +1431,6 @@
// temporal variance.
int short_circuit_low_temp_var;
- // Use modeled (currently CurvFit model) RDCost for fast non-RD mode
- int use_modeled_non_rd_cost;
-
// Reuse inter prediction in fast non-rd mode.
int reuse_inter_pred_nonrd;
@@ -1459,7 +1464,7 @@
int check_scene_detection;
// For nonrd mode: Prefer larger partition blks in variance based partitioning
- // 0: disabled, 1-4: increasing aggressiveness
+ // 0: disabled, 1-3: increasing aggressiveness
int prefer_large_partition_blocks;
// uses results of temporal noise estimate
@@ -1508,8 +1513,18 @@
// variance wrt LAST reference.
int prune_inter_modes_using_temp_var;
- // Force half_pel at block level.
- int force_half_pel_block;
+ // Reduce MV precision to halfpel for higher int MV value & frame-level motion
+ // 0: disabled
+ // 1: Reduce precision to halfpel, fullpel based on conservative thresholds
+ // 2: Reduce precision to halfpel using more aggressive thresholds
+ int reduce_mv_pel_precision_highmotion;
+
+ // Reduce MV precision for low complexity blocks
+ // 0: disabled
+ // 1: Reduce the mv resolution for zero mv if the variance is low
+ // 2: Switch to halfpel, fullpel based on low block spatial-temporal
+ // complexity.
+ int reduce_mv_pel_precision_lowcomplex;
// Prune intra mode evaluation in inter frames based on mv range.
BLOCK_SIZE prune_intra_mode_based_on_mv_range;
@@ -1529,6 +1544,7 @@
int gf_refresh_based_on_qp;
// Temporal filtering
+ // The value can be 1 or 2, which indicates the threshold to use.
int use_rtc_tf;
// Prune the use of the identity transform in nonrd_pickmode,
@@ -1543,6 +1559,9 @@
// For nonrd: early exit out of variance partition that sets the
// block size to superblock size, and sets mode to zeromv-last skip.
+ // 0: disabled
+ // 1: zeromv-skip is enabled at SB level only
+ // 2: zeromv-skip is enabled at SB level and coding block level
int part_early_exit_zeromv;
// Early terminate inter mode search based on sse in non-rd path.
@@ -1554,15 +1573,9 @@
// Enable/disable partition direct merging.
int partition_direct_merging;
- // SAD based compound mode pruning
- int sad_based_comp_prune;
-
// Level of aggressiveness for obtaining tx size based on qstep
int tx_size_level_based_on_qstep;
- // Reduce the mv resolution for zero mv if the variance is low.
- bool reduce_zeromv_mvres;
-
// Avoid the partitioning of a 16x16 block in variance based partitioning
// (VBP) by making use of minimum and maximum sub-block variances.
// For allintra encode, this speed feature reduces instruction count by 5.39%
@@ -1573,22 +1586,40 @@
// gain of 0.78%.
bool vbp_prune_16x16_split_using_min_max_sub_blk_var;
- // A qindex threshold that determines whether to use qindex based
- // CDEF filter strength estimation for screen content types.
- // This speed feature has a substantial gain on coding metrics,
- // with moderate increased encoding time.
- // Set to zero to turn off this speed feature.
+ // A qindex threshold that determines whether to use qindex based CDEF filter
+ // strength estimation for screen content types. The strength estimation model
+ // used for screen contents prefers to allow cdef filtering for more frames.
+ // This sf is used to limit the frames which go through cdef filtering and
+ // following explains the setting of the same.
+ // MAXQ (255): This disables the usage of this sf. Here, frame does not use a
+ // screen content model thus reduces the number of frames that go through cdef
+ // filtering.
+ // MINQ (0): Frames always use screen content model thus increasing the number
+ // of frames that go through cdef filtering.
+ // This speed feature has a substantial gain on coding metrics, with moderate
+ // increase encoding time. Select threshold based on speed vs quality
+ // trade-off.
int screen_content_cdef_filter_qindex_thresh;
- // Prunes global_globalmv search if its variance is \gt the globalmv's
- // variance.
- bool prune_global_globalmv_with_zeromv;
+ // Prune compound mode if its variance is higher than the variance of single
+ // modes.
+ bool prune_compoundmode_with_singlecompound_var;
// Allow mode cost update at frame level every couple frames. This
// overrides the command line setting --mode-cost-upd-freq=3 (never update
// except on key frame and first delta).
bool frame_level_mode_cost_update;
+ // Prune H_PRED during intra mode evaluation in the nonrd path based on best
+ // mode so far.
+ //
+ // For allintra encode, this speed feature reduces instruction count by 1.10%
+ // for speed 9 with coding performance change less than 0.04%.
+ // For AVIF image encode, this speed feature reduces encode time by 1.03% for
+ // speed 9 on a typical image dataset with coding performance change less than
+ // 0.08%.
+ bool prune_h_pred_using_best_mode_so_far;
+
// If compound is enabled, and the current block size is \geq BLOCK_16X16,
// limit the compound modes to GLOBAL_GLOBALMV. This does not apply to the
// base layer of svc.
@@ -1596,6 +1627,51 @@
// Allow for disabling cdf update for non reference frames in svc mode.
bool disable_cdf_update_non_reference_frame;
+
+ // Prune compound modes if the single modes variances do not perform well.
+ bool prune_compoundmode_with_singlemode_var;
+
+ // Skip searching all compound mode if the variance of single_mode residue is
+ // sufficiently low.
+ bool skip_compound_based_on_var;
+
+ // In multi-threaded encoding, enable top right dependency wait of threads at
+ // mi level. Enabling this speed feature has the following assumptions :
+ // 1. Intra modes which use the top right block pixels are disabled.
+ // 2. After the final encoding of the bottom left block in a superblock is
+ // complete, the following members of the structure MB_MODE_INFO of the
+ // already encoded bottom left block are not changed.
+ // i. mode vii. seg_id_predicted
+ // ii. mv[2] viii. ref_mv_idx
+ // iii. ref_frame[2] ix. skip_mode
+ // iv. use_intrabc x. comp_group_idx
+ // v. bsize xi. compound_idx
+ // vi. segment_id xii. use_wedge_interintra
+ // Variables (i) to (v) of the top-right block are required for reference
+ // mv list population. As 'use_intrabc' is a bit-field member and when
+ // accessing it a thread may inadvertently access adjacent bit-fields
+ // it is required that variables (vi) to (xii) are also not updated
+ // after encoding of the bottom left block within a superblock is complete.
+ // Breaking any of the above assumptions may result in thread sanitizer data
+ // race.
+ bool top_right_sync_wait_in_mis;
+
+ // Sets force_zeromv_skip based on the source sad available. Aggressiveness
+ // increases with increase in the level set for speed feature.
+ // 0: No setting
+ // 1: If source sad is kZeroSad
+ // 2: If source sad <= kVeryLowSad
+ int set_zeromv_skip_based_on_source_sad;
+
+ // Downgrades the subpel search to av1_find_best_sub_pixel_tree_pruned_more
+ // when either the fullpel search performed well, or when zeromv has low sad.
+ bool use_adaptive_subpel_search;
+
+ // A flag used in RTC case to control frame_refs_short_signaling. Note that
+ // the final decision is made in check_frame_refs_short_signaling(). The flag
+ // can only be turned on when res < 360p and speed >= 9, in which case only
+ // LAST and GOLDEN ref frames are used now.
+ bool enable_ref_short_signaling;
} REAL_TIME_SPEED_FEATURES;
/*!\endcond */
@@ -1686,7 +1762,7 @@
* \param[in] cpi Top - level encoder instance structure
* \param[in] speed Speed setting passed in from the command line
*
- * \return No return value but configures the various speed trade off flags
+ * \remark No return value but configures the various speed trade off flags
* based on the passed in speed setting. (Higher speed gives lower
* quality)
*/
@@ -1700,7 +1776,7 @@
* \param[in] cpi Top - level encoder instance structure
* \param[in] speed Speed setting passed in from the command line
*
- * \return No return value but configures the various speed trade off flags
+ * \remark No return value but configures the various speed trade off flags
* based on the passed in speed setting and frame size. (Higher speed
* corresponds to lower quality)
*/
@@ -1713,7 +1789,7 @@
* \param[in] cpi Top - level encoder instance structure
* \param[in] speed Speed setting passed in from the command line
*
- * \return No return value but configures the various speed trade off flags
+ * \remark No return value but configures the various speed trade off flags
* based on the passed in speed setting and current frame's Q index.
* (Higher speed corresponds to lower quality)
*/
diff --git a/av1/encoder/svc_layercontext.c b/av1/encoder/svc_layercontext.c
index d5f5353..2575b1b 100644
--- a/av1/encoder/svc_layercontext.c
+++ b/av1/encoder/svc_layercontext.c
@@ -79,9 +79,18 @@
if (svc->number_spatial_layers == 3) {
svc->downsample_filter_type[0] = EIGHTTAP_SMOOTH;
}
- svc->ref_frame_comp[0] = 0;
- svc->ref_frame_comp[1] = 0;
- svc->ref_frame_comp[2] = 0;
+}
+
+void av1_alloc_layer_context(AV1_COMP *cpi, int num_layers) {
+ AV1_COMMON *const cm = &cpi->common;
+ SVC *const svc = &cpi->svc;
+ if (svc->layer_context == NULL || svc->num_allocated_layers < num_layers) {
+ aom_free(svc->layer_context);
+ CHECK_MEM_ERROR(
+ cm, svc->layer_context,
+ (LAYER_CONTEXT *)aom_calloc(num_layers, sizeof(*svc->layer_context)));
+ svc->num_allocated_layers = num_layers;
+ }
}
// Update the layer context from a change_config() call.
@@ -166,14 +175,15 @@
}
static AOM_INLINE bool check_ref_is_low_spatial_res_super_frame(
- int ref_frame, const SVC *svc) {
- int ref_frame_idx = svc->ref_idx[ref_frame - 1];
+ int ref_frame, const SVC *svc, const RTC_REF *rtc_ref) {
+ int ref_frame_idx = rtc_ref->ref_idx[ref_frame - 1];
return svc->buffer_time_index[ref_frame_idx] == svc->current_superframe &&
svc->buffer_spatial_layer[ref_frame_idx] <= svc->spatial_layer_id - 1;
}
void av1_restore_layer_context(AV1_COMP *const cpi) {
SVC *const svc = &cpi->svc;
+ RTC_REF *const rtc_ref = &cpi->rtc_ref;
const AV1_COMMON *const cm = &cpi->common;
LAYER_CONTEXT *const lc = get_layer_context(cpi);
const int old_frame_since_key = cpi->rc.frames_since_key;
@@ -207,14 +217,14 @@
// This is to skip searching mv for that reference if it was last
// refreshed (i.e., buffer slot holding that reference was refreshed) on the
// previous spatial layer(s) at the same time (current_superframe).
- if (svc->set_ref_frame_config && svc->force_zero_mode_spatial_ref) {
- if (check_ref_is_low_spatial_res_super_frame(LAST_FRAME, svc)) {
+ if (rtc_ref->set_ref_frame_config && svc->force_zero_mode_spatial_ref) {
+ if (check_ref_is_low_spatial_res_super_frame(LAST_FRAME, svc, rtc_ref)) {
svc->skip_mvsearch_last = 1;
}
- if (check_ref_is_low_spatial_res_super_frame(GOLDEN_FRAME, svc)) {
+ if (check_ref_is_low_spatial_res_super_frame(GOLDEN_FRAME, svc, rtc_ref)) {
svc->skip_mvsearch_gf = 1;
}
- if (check_ref_is_low_spatial_res_super_frame(ALTREF_FRAME, svc)) {
+ if (check_ref_is_low_spatial_res_super_frame(ALTREF_FRAME, svc, rtc_ref)) {
svc->skip_mvsearch_altref = 1;
}
}
@@ -250,10 +260,10 @@
svc->buffer_time_index[i] = svc->current_superframe;
svc->buffer_spatial_layer[i] = svc->spatial_layer_id;
}
- } else if (cpi->svc.set_ref_frame_config) {
+ } else if (cpi->rtc_ref.set_ref_frame_config) {
for (unsigned int i = 0; i < INTER_REFS_PER_FRAME; i++) {
- int ref_frame_map_idx = svc->ref_idx[i];
- if (cpi->svc.refresh[ref_frame_map_idx]) {
+ int ref_frame_map_idx = cpi->rtc_ref.ref_idx[i];
+ if (cpi->rtc_ref.refresh[ref_frame_map_idx]) {
svc->buffer_time_index[ref_frame_map_idx] = svc->current_superframe;
svc->buffer_spatial_layer[ref_frame_map_idx] = svc->spatial_layer_id;
}
@@ -361,42 +371,43 @@
// spatial and temporal layers, and the ksvc_fixed_mode.
void av1_set_svc_fixed_mode(AV1_COMP *const cpi) {
SVC *const svc = &cpi->svc;
+ RTC_REF *const rtc_ref = &cpi->rtc_ref;
int i;
assert(svc->use_flexible_mode == 0);
// Fixed SVC mode only supports at most 3 spatial or temporal layers.
assert(svc->number_spatial_layers >= 1 && svc->number_spatial_layers <= 3 &&
svc->number_temporal_layers >= 1 && svc->number_temporal_layers <= 3);
- svc->set_ref_frame_config = 1;
+ rtc_ref->set_ref_frame_config = 1;
int superframe_cnt = svc->current_superframe;
// Set the reference map buffer idx for the 7 references:
// LAST_FRAME (0), LAST2_FRAME(1), LAST3_FRAME(2), GOLDEN_FRAME(3),
// BWDREF_FRAME(4), ALTREF2_FRAME(5), ALTREF_FRAME(6).
- for (i = 0; i < INTER_REFS_PER_FRAME; i++) svc->ref_idx[i] = i;
- for (i = 0; i < INTER_REFS_PER_FRAME; i++) svc->reference[i] = 0;
- for (i = 0; i < REF_FRAMES; i++) svc->refresh[i] = 0;
+ for (i = 0; i < INTER_REFS_PER_FRAME; i++) rtc_ref->ref_idx[i] = i;
+ for (i = 0; i < INTER_REFS_PER_FRAME; i++) rtc_ref->reference[i] = 0;
+ for (i = 0; i < REF_FRAMES; i++) rtc_ref->refresh[i] = 0;
// Always reference LAST, and reference GOLDEN on SL > 0.
// For KSVC: GOLDEN reference will be removed on INTER_FRAMES later
// when frame_type is set.
- svc->reference[SVC_LAST_FRAME] = 1;
- if (svc->spatial_layer_id > 0) svc->reference[SVC_GOLDEN_FRAME] = 1;
+ rtc_ref->reference[SVC_LAST_FRAME] = 1;
+ if (svc->spatial_layer_id > 0) rtc_ref->reference[SVC_GOLDEN_FRAME] = 1;
if (svc->temporal_layer_id == 0) {
// Base temporal layer.
if (svc->spatial_layer_id == 0) {
// Set all buffer_idx to 0. Update slot 0 (LAST).
- for (i = 0; i < INTER_REFS_PER_FRAME; i++) svc->ref_idx[i] = 0;
- svc->refresh[0] = 1;
+ for (i = 0; i < INTER_REFS_PER_FRAME; i++) rtc_ref->ref_idx[i] = 0;
+ rtc_ref->refresh[0] = 1;
} else if (svc->spatial_layer_id == 1) {
// Set buffer_idx for LAST to slot 1, GOLDEN (and all other refs) to
// slot 0. Update slot 1 (LAST).
- for (i = 0; i < INTER_REFS_PER_FRAME; i++) svc->ref_idx[i] = 0;
- svc->ref_idx[SVC_LAST_FRAME] = 1;
- svc->refresh[1] = 1;
+ for (i = 0; i < INTER_REFS_PER_FRAME; i++) rtc_ref->ref_idx[i] = 0;
+ rtc_ref->ref_idx[SVC_LAST_FRAME] = 1;
+ rtc_ref->refresh[1] = 1;
} else if (svc->spatial_layer_id == 2) {
// Set buffer_idx for LAST to slot 2, GOLDEN (and all other refs) to
// slot 1. Update slot 2 (LAST).
- for (i = 0; i < INTER_REFS_PER_FRAME; i++) svc->ref_idx[i] = 1;
- svc->ref_idx[SVC_LAST_FRAME] = 2;
- svc->refresh[2] = 1;
+ for (i = 0; i < INTER_REFS_PER_FRAME; i++) rtc_ref->ref_idx[i] = 1;
+ rtc_ref->ref_idx[SVC_LAST_FRAME] = 2;
+ rtc_ref->refresh[2] = 1;
}
} else if (svc->temporal_layer_id == 2 && (superframe_cnt - 1) % 4 == 0) {
// First top temporal enhancement layer.
@@ -404,27 +415,27 @@
// Reference LAST (slot 0).
// Set GOLDEN to slot 3 and update slot 3.
// Set all other buffer_idx to slot 0.
- for (i = 0; i < INTER_REFS_PER_FRAME; i++) svc->ref_idx[i] = 0;
+ for (i = 0; i < INTER_REFS_PER_FRAME; i++) rtc_ref->ref_idx[i] = 0;
if (svc->spatial_layer_id < svc->number_spatial_layers - 1) {
- svc->ref_idx[SVC_GOLDEN_FRAME] = 3;
- svc->refresh[3] = 1;
+ rtc_ref->ref_idx[SVC_GOLDEN_FRAME] = 3;
+ rtc_ref->refresh[3] = 1;
}
} else if (svc->spatial_layer_id == 1) {
// Reference LAST and GOLDEN. Set buffer_idx for LAST to slot 1,
// GOLDEN (and all other refs) to slot 3.
// Set LAST2 to slot 4 and Update slot 4.
- for (i = 0; i < INTER_REFS_PER_FRAME; i++) svc->ref_idx[i] = 3;
- svc->ref_idx[SVC_LAST_FRAME] = 1;
+ for (i = 0; i < INTER_REFS_PER_FRAME; i++) rtc_ref->ref_idx[i] = 3;
+ rtc_ref->ref_idx[SVC_LAST_FRAME] = 1;
if (svc->spatial_layer_id < svc->number_spatial_layers - 1) {
- svc->ref_idx[SVC_LAST2_FRAME] = 4;
- svc->refresh[4] = 1;
+ rtc_ref->ref_idx[SVC_LAST2_FRAME] = 4;
+ rtc_ref->refresh[4] = 1;
}
} else if (svc->spatial_layer_id == 2) {
// Reference LAST and GOLDEN. Set buffer_idx for LAST to slot 2,
// GOLDEN (and all other refs) to slot 4.
// No update.
- for (i = 0; i < INTER_REFS_PER_FRAME; i++) svc->ref_idx[i] = 4;
- svc->ref_idx[SVC_LAST_FRAME] = 2;
+ for (i = 0; i < INTER_REFS_PER_FRAME; i++) rtc_ref->ref_idx[i] = 4;
+ rtc_ref->ref_idx[SVC_LAST_FRAME] = 2;
}
} else if (svc->temporal_layer_id == 1) {
// Middle temporal enhancement layer.
@@ -432,30 +443,30 @@
// Reference LAST.
// Set all buffer_idx to 0.
// Set GOLDEN to slot 5 and update slot 5.
- for (i = 0; i < INTER_REFS_PER_FRAME; i++) svc->ref_idx[i] = 0;
+ for (i = 0; i < INTER_REFS_PER_FRAME; i++) rtc_ref->ref_idx[i] = 0;
if (svc->temporal_layer_id < svc->number_temporal_layers - 1) {
- svc->ref_idx[SVC_GOLDEN_FRAME] = 5;
- svc->refresh[5] = 1;
+ rtc_ref->ref_idx[SVC_GOLDEN_FRAME] = 5;
+ rtc_ref->refresh[5] = 1;
}
} else if (svc->spatial_layer_id == 1) {
// Reference LAST and GOLDEN. Set buffer_idx for LAST to slot 1,
// GOLDEN (and all other refs) to slot 5.
// Set LAST3 to slot 6 and update slot 6.
- for (i = 0; i < INTER_REFS_PER_FRAME; i++) svc->ref_idx[i] = 5;
- svc->ref_idx[SVC_LAST_FRAME] = 1;
+ for (i = 0; i < INTER_REFS_PER_FRAME; i++) rtc_ref->ref_idx[i] = 5;
+ rtc_ref->ref_idx[SVC_LAST_FRAME] = 1;
if (svc->temporal_layer_id < svc->number_temporal_layers - 1) {
- svc->ref_idx[SVC_LAST3_FRAME] = 6;
- svc->refresh[6] = 1;
+ rtc_ref->ref_idx[SVC_LAST3_FRAME] = 6;
+ rtc_ref->refresh[6] = 1;
}
} else if (svc->spatial_layer_id == 2) {
// Reference LAST and GOLDEN. Set buffer_idx for LAST to slot 2,
// GOLDEN (and all other refs) to slot 6.
// Set LAST3 to slot 7 and update slot 7.
- for (i = 0; i < INTER_REFS_PER_FRAME; i++) svc->ref_idx[i] = 6;
- svc->ref_idx[SVC_LAST_FRAME] = 2;
+ for (i = 0; i < INTER_REFS_PER_FRAME; i++) rtc_ref->ref_idx[i] = 6;
+ rtc_ref->ref_idx[SVC_LAST_FRAME] = 2;
if (svc->temporal_layer_id < svc->number_temporal_layers - 1) {
- svc->ref_idx[SVC_LAST3_FRAME] = 7;
- svc->refresh[7] = 1;
+ rtc_ref->ref_idx[SVC_LAST3_FRAME] = 7;
+ rtc_ref->refresh[7] = 1;
}
}
} else if (svc->temporal_layer_id == 2 && (superframe_cnt - 3) % 4 == 0) {
@@ -464,28 +475,28 @@
// Set LAST to slot 5 and reference LAST.
// Set GOLDEN to slot 3 and update slot 3.
// Set all other buffer_idx to 0.
- for (i = 0; i < INTER_REFS_PER_FRAME; i++) svc->ref_idx[i] = 0;
- svc->ref_idx[SVC_LAST_FRAME] = 5;
+ for (i = 0; i < INTER_REFS_PER_FRAME; i++) rtc_ref->ref_idx[i] = 0;
+ rtc_ref->ref_idx[SVC_LAST_FRAME] = 5;
if (svc->spatial_layer_id < svc->number_spatial_layers - 1) {
- svc->ref_idx[SVC_GOLDEN_FRAME] = 3;
- svc->refresh[3] = 1;
+ rtc_ref->ref_idx[SVC_GOLDEN_FRAME] = 3;
+ rtc_ref->refresh[3] = 1;
}
} else if (svc->spatial_layer_id == 1) {
// Reference LAST and GOLDEN. Set buffer_idx for LAST to slot 6,
// GOLDEN to slot 3. Set LAST2 to slot 4 and update slot 4.
- for (i = 0; i < INTER_REFS_PER_FRAME; i++) svc->ref_idx[i] = 0;
- svc->ref_idx[SVC_LAST_FRAME] = 6;
- svc->ref_idx[SVC_GOLDEN_FRAME] = 3;
+ for (i = 0; i < INTER_REFS_PER_FRAME; i++) rtc_ref->ref_idx[i] = 0;
+ rtc_ref->ref_idx[SVC_LAST_FRAME] = 6;
+ rtc_ref->ref_idx[SVC_GOLDEN_FRAME] = 3;
if (svc->spatial_layer_id < svc->number_spatial_layers - 1) {
- svc->ref_idx[SVC_LAST2_FRAME] = 4;
- svc->refresh[4] = 1;
+ rtc_ref->ref_idx[SVC_LAST2_FRAME] = 4;
+ rtc_ref->refresh[4] = 1;
}
} else if (svc->spatial_layer_id == 2) {
// Reference LAST and GOLDEN. Set buffer_idx for LAST to slot 7,
// GOLDEN to slot 4. No update.
- for (i = 0; i < INTER_REFS_PER_FRAME; i++) svc->ref_idx[i] = 0;
- svc->ref_idx[SVC_LAST_FRAME] = 7;
- svc->ref_idx[SVC_GOLDEN_FRAME] = 4;
+ for (i = 0; i < INTER_REFS_PER_FRAME; i++) rtc_ref->ref_idx[i] = 0;
+ rtc_ref->ref_idx[SVC_LAST_FRAME] = 7;
+ rtc_ref->ref_idx[SVC_GOLDEN_FRAME] = 4;
}
}
}
diff --git a/av1/encoder/svc_layercontext.h b/av1/encoder/svc_layercontext.h
index e3c2653..a27d768 100644
--- a/av1/encoder/svc_layercontext.h
+++ b/av1/encoder/svc_layercontext.h
@@ -90,22 +90,11 @@
int temporal_layer_id;
int number_spatial_layers;
int number_temporal_layers;
- int set_ref_frame_config;
- int non_reference_frame;
int use_flexible_mode;
int ksvc_fixed_mode;
- int ref_frame_comp[3];
/*!\endcond */
- /*!
- * LAST_FRAME (0), LAST2_FRAME(1), LAST3_FRAME(2), GOLDEN_FRAME(3),
- * BWDREF_FRAME(4), ALTREF2_FRAME(5), ALTREF_FRAME(6).
- */
- int reference[INTER_REFS_PER_FRAME];
/*!\cond */
- int ref_idx[INTER_REFS_PER_FRAME];
- int refresh[REF_FRAMES];
- int gld_idx_1layer;
double base_framerate;
unsigned int current_superframe;
unsigned int buffer_time_index[REF_FRAMES];
@@ -123,7 +112,12 @@
/*!
* Layer context used for rate control in CBR mode.
*/
- LAYER_CONTEXT layer_context[AOM_MAX_LAYERS];
+ LAYER_CONTEXT *layer_context;
+
+ /*!
+ * Number of layers allocated for layer_context.
+ */
+ int num_allocated_layers;
/*!
* EIGHTTAP_SMOOTH or BILINEAR
@@ -152,10 +146,23 @@
*
* \param[in] cpi Top level encoder structure
*
- * \return Nothing returned. Set cpi->svc.
+ * \remark Nothing returned. Set cpi->svc.
*/
void av1_init_layer_context(struct AV1_COMP *const cpi);
+/*!\brief Allocate layer context data.
+ *
+ * \ingroup SVC
+ * \callgraph
+ * \callergraph
+ *
+ * \param[in] cpi Top level encoder structure
+ * \param[in] num_layers Number of layers to be allocated
+ *
+ * \remark Nothing returned. Allocates memory for cpi->svc.layer_context.
+ */
+void av1_alloc_layer_context(struct AV1_COMP *cpi, int num_layers);
+
/*!\brief Update the layer context from a change_config() call.
*
* \ingroup SVC
@@ -165,7 +172,7 @@
* \param[in] cpi Top level encoder structure
* \param[in] target_bandwidth Total target bandwidth
*
- * \return Nothing returned. Buffer level for each layer is set.
+ * \remark Nothing returned. Buffer level for each layer is set.
*/
void av1_update_layer_context_change_config(struct AV1_COMP *const cpi,
const int64_t target_bandwidth);
@@ -179,7 +186,7 @@
*
* \param[in] cpi Top level encoder structure
*
- * \return Nothing returned. Frame related quantities for current temporal
+ * \remark Nothing returned. Frame related quantities for current temporal
layer are updated.
*/
void av1_update_temporal_layer_framerate(struct AV1_COMP *const cpi);
@@ -193,7 +200,7 @@
*
* \param[in] cpi Top level encoder structure
*
- * \return Nothing returned. Layer context for current layer is set.
+ * \remark Nothing returned. Layer context for current layer is set.
*/
void av1_restore_layer_context(struct AV1_COMP *const cpi);
@@ -204,8 +211,6 @@
* \callergraph
*
* \param[in] cpi Top level encoder structure
- *
- * \return Nothing returned.
*/
void av1_save_layer_context(struct AV1_COMP *const cpi);
@@ -216,8 +221,6 @@
* \callergraph
*
* \param[in] cpi Top level encoder structure
- *
- * \return Nothing returned.
*/
void av1_free_svc_cyclic_refresh(struct AV1_COMP *const cpi);
@@ -229,8 +232,6 @@
*
* \param[in] cpi Top level encoder structure
* \param[in] is_key Whether current layer is key frame
- *
- * \return Nothing returned.
*/
void av1_svc_reset_temporal_layers(struct AV1_COMP *const cpi, int is_key);
@@ -241,8 +242,6 @@
* \callergraph
*
* \param[in] cpi Top level encoder structure
- *
- * \return Nothing returned.
*/
void av1_one_pass_cbr_svc_start_layer(struct AV1_COMP *const cpi);
@@ -268,7 +267,7 @@
* \param[in] width_out Output width, scaled for current layer
* \param[in] height_out Output height, scaled for current layer
*
- * \return Nothing is returned. Instead the scaled width and height are set.
+ * \remark Nothing is returned. Instead the scaled width and height are set.
*/
void av1_get_layer_resolution(const int width_org, const int height_org,
const int num, const int den, int *width_out,
diff --git a/av1/encoder/temporal_filter.c b/av1/encoder/temporal_filter.c
index cc7b49f..62a4d3b 100644
--- a/av1/encoder/temporal_filter.c
+++ b/av1/encoder/temporal_filter.c
@@ -81,7 +81,7 @@
* \param[out] subblock_mses Pointer to the search errors (MSE) for 4
* sub-blocks
*
- * \return Nothing will be returned. Results are saved in subblock_mvs and
+ * \remark Nothing will be returned. Results are saved in subblock_mvs and
* subblock_mses
*/
static void tf_motion_search(AV1_COMP *cpi, MACROBLOCK *mb,
@@ -323,7 +323,7 @@
* order)
* \param[out] pred Pointer to the predictor to be built
*
- * \return Nothing returned, But the contents of `pred` will be modified
+ * \remark Nothing returned, But the contents of `pred` will be modified
*/
static void tf_build_predictor(const YV12_BUFFER_CONFIG *ref_frame,
const MACROBLOCKD *mbd,
@@ -551,7 +551,7 @@
* \param[out] count Pointer to the pixel-wise counter for
* filtering
*
- * \return Nothing returned, But the contents of `accum`, `pred` and 'count'
+ * \remark Nothing returned, But the contents of `accum`, `pred` and 'count'
* will be modified
*/
void av1_apply_temporal_filter_c(
@@ -734,7 +734,7 @@
* \param[in] count Pointer to the pre-computed count
* \param[out] result_buffer Pointer to result buffer
*
- * \return Nothing returned, but the content to which `result_buffer` pointer
+ * \remark Nothing returned, but the content to which `result_buffer` pointer
* will be modified
*/
static void tf_normalize_filtered_frame(
@@ -914,7 +914,7 @@
* \ingroup src_frame_proc
* \param[in] cpi Top level encoder instance structure
*
- * \return Nothing will be returned, but the contents of td->diff will be
+ * \remark Nothing will be returned, but the contents of td->diff will be
modified.
*/
static void tf_do_filtering(AV1_COMP *cpi) {
@@ -949,7 +949,7 @@
* in the lookahead buffer cpi->lookahead
* \param[in] gf_frame_index GOP index
*
- * \return Nothing will be returned. But the fields `frames`, `num_frames`,
+ * \remark Nothing will be returned. But the fields `frames`, `num_frames`,
* `filter_frame_idx` and `noise_levels` will be updated in cpi->tf_ctx.
*/
static void tf_setup_filtering_buffer(AV1_COMP *cpi,
@@ -1295,17 +1295,6 @@
"Failed to allocate tf_info");
}
}
-
- ret = aom_realloc_frame_buffer(
- &tf_info->tf_buf_second_arf, oxcf->frm_dim_cfg.width,
- oxcf->frm_dim_cfg.height, seq_params->subsampling_x,
- seq_params->subsampling_y, seq_params->use_highbitdepth,
- cpi->oxcf.border_in_pixels, cm->features.byte_alignment, NULL, NULL, NULL,
- cpi->oxcf.tool_cfg.enable_global_motion, 0);
- if (ret) {
- aom_internal_error(cm->error, AOM_CODEC_MEM_ERROR,
- "Failed to allocate tf_info");
- }
}
void av1_tf_info_free(TEMPORAL_FILTER_INFO *tf_info) {
diff --git a/av1/encoder/temporal_filter.h b/av1/encoder/temporal_filter.h
index 633dbe4..725bd86 100644
--- a/av1/encoder/temporal_filter.h
+++ b/av1/encoder/temporal_filter.h
@@ -287,7 +287,7 @@
* \param[in] mb_row Macroblock row to be filtered
filtering
*
-* \return Nothing will be returned, but the contents of td->diff will be
+* \remark Nothing will be returned, but the contents of td->diff will be
modified.
*/
void av1_tf_do_filtering_row(struct AV1_COMP *cpi, struct ThreadData *td,
diff --git a/av1/encoder/thirdpass.c b/av1/encoder/thirdpass.c
index d526554..cbd9a69 100644
--- a/av1/encoder/thirdpass.c
+++ b/av1/encoder/thirdpass.c
@@ -8,7 +8,9 @@
* Media Patent License 1.0 was not distributed with this source code in the
* PATENTS file, you can obtain it at www.aomedia.org/license/patent.
*/
+#include "av1/encoder/thirdpass.h"
+#if CONFIG_THREE_PASS && CONFIG_AV1_DECODER
#include "aom/aom_codec.h"
#include "aom/aomdx.h"
#include "aom_dsp/psnr.h"
@@ -16,14 +18,9 @@
#include "av1/av1_iface_common.h"
#include "av1/encoder/encoder.h"
#include "av1/encoder/firstpass.h"
-#include "av1/encoder/thirdpass.h"
#include "av1/common/blockd.h"
-
-#if CONFIG_THREE_PASS
#include "common/ivfdec.h"
-#endif
-#if CONFIG_THREE_PASS
static void setup_two_pass_stream_input(
struct AvxInputContext **input_ctx_ptr, const char *input_file_name,
struct aom_internal_error_info *err_info) {
@@ -64,7 +61,6 @@
ctx->err_info);
}
-#if CONFIG_AV1_DECODER
if (!ctx->decoder.iface) {
aom_codec_iface_t *decoder_iface = &aom_codec_av1_inspect_algo;
if (aom_codec_dec_init(&ctx->decoder, decoder_iface, NULL, 0)) {
@@ -72,19 +68,12 @@
"Failed to initialize decoder.");
}
}
-#else
- aom_internal_error(ctx->err_info, AOM_CODEC_ERROR,
- "To utilize three-pass encoding, libaom must be built "
- "with CONFIG_AV1_DECODER=1.");
-#endif
}
-#endif // CONFIG_THREE_PASS
// Return 0: success
// 1: cannot read because this is end of file
// -1: failure to read the frame
static int read_frame(THIRD_PASS_DEC_CTX *ctx) {
-#if CONFIG_THREE_PASS
if (!ctx->input_ctx || !ctx->decoder.iface) {
init_third_pass(ctx);
}
@@ -101,10 +90,7 @@
ctx->end_frame = ctx->frame + ctx->bytes_in_buffer;
ctx->have_frame = 1;
}
-#else
- aom_internal_error(ctx->err_info, AOM_CODEC_ERROR,
- "Cannot parse bitstream without CONFIG_THREE_PASS.");
-#endif
+
Av1DecodeReturn adr;
if (aom_codec_decode(&ctx->decoder, ctx->frame,
(unsigned int)ctx->bytes_in_buffer,
@@ -404,10 +390,8 @@
if (ctx->decoder.iface) {
aom_codec_destroy(&ctx->decoder);
}
-#if CONFIG_THREE_PASS
if (ctx->input_ctx && ctx->input_ctx->file) fclose(ctx->input_ctx->file);
aom_free(ctx->input_ctx);
-#endif
if (ctx->buf) free(ctx->buf);
for (int i = 0; i < MAX_THIRD_PASS_BUF; i++) {
free_frame_info(&ctx->frame_info[i]);
@@ -706,6 +690,119 @@
return corner_mi->partition;
}
+#else // !(CONFIG_THREE_PASS && CONFIG_AV1_DECODER)
+void av1_init_thirdpass_ctx(AV1_COMMON *cm, THIRD_PASS_DEC_CTX **ctx,
+ const char *file) {
+ (void)ctx;
+ (void)file;
+ aom_internal_error(cm->error, AOM_CODEC_ERROR,
+ "To utilize three-pass encoding, libaom must be built "
+ "with CONFIG_THREE_PASS=1 & CONFIG_AV1_DECODER=1.");
+}
+
+void av1_free_thirdpass_ctx(THIRD_PASS_DEC_CTX *ctx) { (void)ctx; }
+
+void av1_set_gop_third_pass(THIRD_PASS_DEC_CTX *ctx) { (void)ctx; }
+
+void av1_pop_third_pass_info(THIRD_PASS_DEC_CTX *ctx) { (void)ctx; }
+
+void av1_open_second_pass_log(struct AV1_COMP *cpi, int is_read) {
+ (void)cpi;
+ (void)is_read;
+}
+
+void av1_close_second_pass_log(struct AV1_COMP *cpi) { (void)cpi; }
+
+void av1_write_second_pass_gop_info(struct AV1_COMP *cpi) { (void)cpi; }
+
+void av1_write_second_pass_per_frame_info(struct AV1_COMP *cpi, int gf_index) {
+ (void)cpi;
+ (void)gf_index;
+}
+
+void av1_read_second_pass_gop_info(FILE *second_pass_log_stream,
+ THIRD_PASS_GOP_INFO *gop_info,
+ struct aom_internal_error_info *error) {
+ (void)second_pass_log_stream;
+ (void)gop_info;
+ (void)error;
+}
+
+void av1_read_second_pass_per_frame_info(
+ FILE *second_pass_log_stream, THIRD_PASS_FRAME_INFO *frame_info_arr,
+ int frame_info_count, struct aom_internal_error_info *error) {
+ (void)second_pass_log_stream;
+ (void)frame_info_arr;
+ (void)frame_info_count;
+ (void)error;
+}
+
+int av1_check_use_arf(THIRD_PASS_DEC_CTX *ctx) {
+ (void)ctx;
+ return 1;
+}
+
+void av1_get_third_pass_ratio(THIRD_PASS_DEC_CTX *ctx, int fidx, int fheight,
+ int fwidth, double *ratio_h, double *ratio_w) {
+ (void)ctx;
+ (void)fidx;
+ (void)fheight;
+ (void)fwidth;
+ (void)ratio_h;
+ (void)ratio_w;
+}
+
+THIRD_PASS_MI_INFO *av1_get_third_pass_mi(THIRD_PASS_DEC_CTX *ctx, int fidx,
+ int mi_row, int mi_col,
+ double ratio_h, double ratio_w) {
+ (void)ctx;
+ (void)fidx;
+ (void)mi_row;
+ (void)mi_col;
+ (void)ratio_h;
+ (void)ratio_w;
+ return NULL;
+}
+
+int_mv av1_get_third_pass_adjusted_mv(THIRD_PASS_MI_INFO *this_mi,
+ double ratio_h, double ratio_w,
+ MV_REFERENCE_FRAME frame) {
+ (void)this_mi;
+ (void)ratio_h;
+ (void)ratio_w;
+ (void)frame;
+ int_mv mv;
+ mv.as_int = INVALID_MV;
+ return mv;
+}
+
+BLOCK_SIZE av1_get_third_pass_adjusted_blk_size(THIRD_PASS_MI_INFO *this_mi,
+ double ratio_h,
+ double ratio_w) {
+ (void)this_mi;
+ (void)ratio_h;
+ (void)ratio_w;
+ return BLOCK_INVALID;
+}
+
+void av1_third_pass_get_adjusted_mi(THIRD_PASS_MI_INFO *third_pass_mi,
+ double ratio_h, double ratio_w, int *mi_row,
+ int *mi_col) {
+ (void)third_pass_mi;
+ (void)ratio_h;
+ (void)ratio_w;
+ (void)mi_row;
+ (void)mi_col;
+}
+
+PARTITION_TYPE av1_third_pass_get_sb_part_type(THIRD_PASS_DEC_CTX *ctx,
+ THIRD_PASS_MI_INFO *this_mi) {
+ (void)ctx;
+ (void)this_mi;
+ return PARTITION_INVALID;
+}
+#endif // CONFIG_THREE_PASS && CONFIG_AV1_DECODER
+
#if CONFIG_BITRATE_ACCURACY
static void fwrite_and_check(const void *ptr, size_t size, size_t nmemb,
FILE *stream,
diff --git a/av1/encoder/tpl_model.c b/av1/encoder/tpl_model.c
index b054770..b8366f0 100644
--- a/av1/encoder/tpl_model.c
+++ b/av1/encoder/tpl_model.c
@@ -196,7 +196,7 @@
}
}
-static AOM_INLINE int64_t tpl_get_satd_cost(BitDepthInfo bd_info,
+static AOM_INLINE int32_t tpl_get_satd_cost(BitDepthInfo bd_info,
int16_t *src_diff, int diff_stride,
const uint8_t *src, int src_stride,
const uint8_t *dst, int dst_stride,
@@ -458,8 +458,8 @@
int frame_offset = tpl_data->frame_idx - cpi->gf_frame_index;
- int64_t best_intra_cost = INT64_MAX;
- int64_t intra_cost;
+ int32_t best_intra_cost = INT32_MAX;
+ int32_t intra_cost;
PREDICTION_MODE best_mode = DC_PRED;
int mb_y_offset = mi_row * MI_SIZE * xd->cur_buf->y_stride + mi_col * MI_SIZE;
@@ -606,8 +606,8 @@
int best_rf_idx = -1;
int_mv best_mv[2];
- int64_t inter_cost;
- int64_t best_inter_cost = INT64_MAX;
+ int32_t inter_cost;
+ int32_t best_inter_cost = INT32_MAX;
int rf_idx;
int_mv single_mv[INTER_REFS_PER_FRAME];
@@ -878,7 +878,7 @@
xd->mi[0]->ref_frame[1] = best_rf_idx1 + LAST_FRAME;
}
- if (best_inter_cost < INT64_MAX) {
+ if (best_inter_cost < INT32_MAX) {
xd->mi[0]->mv[0].as_int = best_mv[0].as_int;
xd->mi[0]->mv[1].as_int = best_mv[1].as_int;
const YV12_BUFFER_CONFIG *ref_frame_ptr[2] = {
@@ -894,13 +894,13 @@
qcoeff, dqcoeff, cm, x, ref_frame_ptr, rec_buffer_pool,
rec_stride_pool, tx_size, best_mode, mi_row, mi_col,
use_y_only_rate_distortion, NULL);
- tpl_stats->srcrf_rate = rate_cost << TPL_DEP_COST_SCALE_LOG2;
+ tpl_stats->srcrf_rate = rate_cost;
}
best_intra_cost = AOMMAX(best_intra_cost, 1);
best_inter_cost = AOMMIN(best_intra_cost, best_inter_cost);
- tpl_stats->inter_cost = best_inter_cost << TPL_DEP_COST_SCALE_LOG2;
- tpl_stats->intra_cost = best_intra_cost << TPL_DEP_COST_SCALE_LOG2;
+ tpl_stats->inter_cost = best_inter_cost;
+ tpl_stats->intra_cost = best_intra_cost;
tpl_stats->srcrf_dist = recon_error << TPL_DEP_COST_SCALE_LOG2;
tpl_stats->srcrf_sse = pred_error << TPL_DEP_COST_SCALE_LOG2;
@@ -912,7 +912,8 @@
ref_frame_ptr[0] =
best_mode == NEW_NEWMV
? tpl_data->ref_frame[comp_ref_frames[best_cmp_rf_idx][0]]
- : best_rf_idx >= 0 ? tpl_data->ref_frame[best_rf_idx] : NULL;
+ : best_rf_idx >= 0 ? tpl_data->ref_frame[best_rf_idx]
+ : NULL;
ref_frame_ptr[1] =
best_mode == NEW_NEWMV
? tpl_data->ref_frame[comp_ref_frames[best_cmp_rf_idx][1]]
@@ -923,11 +924,11 @@
use_y_only_rate_distortion, tpl_txfm_stats);
tpl_stats->recrf_dist = recon_error << (TPL_DEP_COST_SCALE_LOG2);
- tpl_stats->recrf_rate = rate_cost << TPL_DEP_COST_SCALE_LOG2;
+ tpl_stats->recrf_rate = rate_cost;
if (!is_inter_mode(best_mode)) {
tpl_stats->srcrf_dist = recon_error << (TPL_DEP_COST_SCALE_LOG2);
- tpl_stats->srcrf_rate = rate_cost << TPL_DEP_COST_SCALE_LOG2;
+ tpl_stats->srcrf_rate = rate_cost;
tpl_stats->srcrf_sse = pred_error << TPL_DEP_COST_SCALE_LOG2;
}
@@ -943,7 +944,7 @@
rec_stride_pool, tx_size, best_mode, mi_row, mi_col,
use_y_only_rate_distortion, NULL);
tpl_stats->cmp_recrf_dist[0] = recon_error << TPL_DEP_COST_SCALE_LOG2;
- tpl_stats->cmp_recrf_rate[0] = rate_cost << TPL_DEP_COST_SCALE_LOG2;
+ tpl_stats->cmp_recrf_rate[0] = rate_cost;
tpl_stats->cmp_recrf_dist[0] =
AOMMAX(tpl_stats->srcrf_dist, tpl_stats->cmp_recrf_dist[0]);
@@ -964,7 +965,7 @@
rec_stride_pool, tx_size, best_mode, mi_row, mi_col,
use_y_only_rate_distortion, NULL);
tpl_stats->cmp_recrf_dist[1] = recon_error << TPL_DEP_COST_SCALE_LOG2;
- tpl_stats->cmp_recrf_rate[1] = rate_cost << TPL_DEP_COST_SCALE_LOG2;
+ tpl_stats->cmp_recrf_rate[1] = rate_cost;
tpl_stats->cmp_recrf_dist[1] =
AOMMAX(tpl_stats->srcrf_dist, tpl_stats->cmp_recrf_dist[1]);
@@ -1098,15 +1099,18 @@
int64_t srcrf_dist = is_compound ? tpl_stats_ptr->cmp_recrf_dist[!ref]
: tpl_stats_ptr->srcrf_dist;
- int64_t srcrf_rate = is_compound ? tpl_stats_ptr->cmp_recrf_rate[!ref]
- : tpl_stats_ptr->srcrf_rate;
+ int64_t srcrf_rate =
+ is_compound
+ ? (tpl_stats_ptr->cmp_recrf_rate[!ref] << TPL_DEP_COST_SCALE_LOG2)
+ : (tpl_stats_ptr->srcrf_rate << TPL_DEP_COST_SCALE_LOG2);
int64_t cur_dep_dist = tpl_stats_ptr->recrf_dist - srcrf_dist;
int64_t mc_dep_dist =
(int64_t)(tpl_stats_ptr->mc_dep_dist *
((double)(tpl_stats_ptr->recrf_dist - srcrf_dist) /
tpl_stats_ptr->recrf_dist));
- int64_t delta_rate = tpl_stats_ptr->recrf_rate - srcrf_rate;
+ int64_t delta_rate =
+ (tpl_stats_ptr->recrf_rate << TPL_DEP_COST_SCALE_LOG2) - srcrf_rate;
int64_t mc_dep_rate =
av1_delta_rate_cost(tpl_stats_ptr->mc_dep_rate, tpl_stats_ptr->recrf_dist,
srcrf_dist, pix_num);
diff --git a/av1/encoder/tpl_model.h b/av1/encoder/tpl_model.h
index b77a19f..ec49ea5 100644
--- a/av1/encoder/tpl_model.h
+++ b/av1/encoder/tpl_model.h
@@ -104,20 +104,20 @@
} TplTxfmStats;
typedef struct TplDepStats {
- int64_t intra_cost;
- int64_t inter_cost;
+ int64_t srcrf_sse;
int64_t srcrf_dist;
int64_t recrf_dist;
int64_t cmp_recrf_dist[2];
- int64_t srcrf_rate;
- int64_t recrf_rate;
- int64_t srcrf_sse;
- int64_t cmp_recrf_rate[2];
int64_t mc_dep_rate;
int64_t mc_dep_dist;
- int_mv mv[INTER_REFS_PER_FRAME];
- int ref_frame_index[2];
int64_t pred_error[INTER_REFS_PER_FRAME];
+ int32_t intra_cost;
+ int32_t inter_cost;
+ int32_t srcrf_rate;
+ int32_t recrf_rate;
+ int32_t cmp_recrf_rate[2];
+ int_mv mv[INTER_REFS_PER_FRAME];
+ int8_t ref_frame_index[2];
} TplDepStats;
typedef struct TplDepFrame {
diff --git a/av1/encoder/tx_search.c b/av1/encoder/tx_search.c
index ea0288b..74c9de2 100644
--- a/av1/encoder/tx_search.c
+++ b/av1/encoder/tx_search.c
@@ -3473,7 +3473,7 @@
assert(level >= 0 && level <= 2);
int model_rate;
int64_t model_dist;
- int model_skip;
+ uint8_t model_skip;
MACROBLOCKD *const xd = &x->e_mbd;
model_rd_sb_fn[MODELRD_TYPE_TX_SEARCH_PRUNE](
cpi, bsize, x, xd, 0, 0, &model_rate, &model_dist, &model_skip, NULL,
diff --git a/av1/encoder/tx_search.h b/av1/encoder/tx_search.h
index e3caf5b..b3689cf 100644
--- a/av1/encoder/tx_search.h
+++ b/av1/encoder/tx_search.h
@@ -89,7 +89,7 @@
* \param[in] rd_stats Pointer to struct to keep track of the RD stats
* \param[in] bsize Current macroblock size
* \param[in] ref_best_rd Best RD cost seen for this block so far
- * \return Nothing is returned. The selected transform size and type will
+ * \remark Nothing is returned. The selected transform size and type will
be saved in the MB_MODE_INFO structure
*/
void av1_pick_recursive_tx_size_type_yrd(const AV1_COMP *cpi, MACROBLOCK *x,
@@ -111,7 +111,7 @@
* \param[in] rd_stats Pointer to struct to keep track of the RD stats
* \param[in] bs Current macroblock size
* \param[in] ref_best_rd Best RD cost seen for this block so far
- * \return Nothing is returned. The selected transform size and type will
+ * \remark Nothing is returned. The selected transform size and type will
be saved in the MB_MODE_INFO structure
*/
void av1_pick_uniform_tx_size_type_yrd(const AV1_COMP *const cpi, MACROBLOCK *x,
@@ -158,7 +158,7 @@
* \param[in] skip_trellis Binary flag indicating if trellis optimization
should be skipped
*
- * \return Nothing is returned. The RD results will be saved in rd_stats.
+ * \remark Nothing is returned. The RD results will be saved in rd_stats.
*/
void av1_txfm_rd_in_plane(MACROBLOCK *x, const AV1_COMP *cpi,
RD_STATS *rd_stats, int64_t ref_best_rd,
diff --git a/av1/encoder/var_based_part.c b/av1/encoder/var_based_part.c
index ab27d1f..6d29bb7 100644
--- a/av1/encoder/var_based_part.c
+++ b/av1/encoder/var_based_part.c
@@ -383,14 +383,16 @@
int highbd_flag,
#endif
int pixels_wide, int pixels_high,
- int is_key_frame) {
+ int is_key_frame,
+ int border_offset_4x4) {
int k;
for (k = 0; k < 4; k++) {
int x4_idx = x8_idx + ((k & 1) << 2);
int y4_idx = y8_idx + ((k >> 1) << 2);
unsigned int sse = 0;
int sum = 0;
- if (x4_idx < pixels_wide && y4_idx < pixels_high) {
+ if (x4_idx < pixels_wide - border_offset_4x4 &&
+ y4_idx < pixels_high - border_offset_4x4) {
int s_avg;
int d_avg = 128;
#if CONFIG_AV1_HIGHBITDEPTH
@@ -429,8 +431,7 @@
}
static AOM_INLINE void tune_thresh_based_on_qindex_window(
- int qindex, int th, int source_sad, int ag_idx, int64_t thresholds[]) {
- const int win = 45;
+ int qindex, int th, int win, int fac, int64_t thresholds[]) {
double weight;
if (qindex < th - win)
@@ -443,7 +444,6 @@
(int)((1 - weight) * (thresholds[1] << 1) + weight * thresholds[1]);
thresholds[2] =
(int)((1 - weight) * (thresholds[2] << 1) + weight * thresholds[2]);
- const int fac = (!ag_idx && source_sad != kLowSad) ? 1 : 2;
thresholds[3] =
(int)((1 - weight) * (thresholds[3] << fac) + weight * thresholds[3]);
}
@@ -451,7 +451,9 @@
static AOM_INLINE void set_vbp_thresholds(AV1_COMP *cpi, int64_t thresholds[],
int q, int content_lowsumdiff,
int source_sad_nonrd,
- int source_sad_rd, int segment_id) {
+ int source_sad_rd, int segment_id,
+ uint64_t blk_sad,
+ int lighting_change) {
AV1_COMMON *const cm = &cpi->common;
const int is_key_frame = frame_is_intra_only(cm);
const int threshold_multiplier = is_key_frame ? 120 : 1;
@@ -510,12 +512,12 @@
else
threshold_base =
scale_part_thresh_content(threshold_base, cpi->oxcf.speed, cm->width,
- cm->height, cpi->svc.non_reference_frame);
+ cm->height, cpi->rtc_ref.non_reference_frame);
#else
// Increase base variance threshold based on content_state/sum_diff level.
threshold_base =
scale_part_thresh_content(threshold_base, cpi->oxcf.speed, cm->width,
- cm->height, cpi->svc.non_reference_frame);
+ cm->height, cpi->rtc_ref.non_reference_frame);
#endif
thresholds[0] = threshold_base >> 1;
thresholds[1] = threshold_base;
@@ -565,11 +567,13 @@
thresholds[2] = (5 * threshold_base) >> 2;
} else if (cm->width < 1920 && cm->height < 1080) {
thresholds[2] = threshold_base << 1;
- } else {
+ } else if (cm->width < 2560 && cm->height < 1440) {
thresholds[2] = (5 * threshold_base) >> 1;
+ } else {
+ thresholds[2] = (7 * threshold_base) >> 1;
}
// Tune thresholds less or more aggressively to prefer larger partitions
- if (cpi->sf.rt_sf.prefer_large_partition_blocks >= 4) {
+ if (cpi->sf.rt_sf.prefer_large_partition_blocks >= 3) {
double weight;
const int win = 20;
if (current_qindex < QINDEX_LARGE_BLOCK_THR - win)
@@ -585,14 +589,23 @@
}
}
if (cm->width * cm->height <= 352 * 288) {
- thresholds[3] = INT32_MAX;
+ thresholds[3] = INT64_MAX;
if (segment_id == 0) {
thresholds[1] <<= 2;
- thresholds[2] <<= (source_sad_nonrd == kLowSad) ? 5 : 4;
+ thresholds[2] <<= (source_sad_nonrd <= kLowSad) ? 5 : 4;
} else {
thresholds[1] <<= 1;
thresholds[2] <<= 3;
}
+ // Allow for split to 8x8 for superblocks where part of it has
+ // moving boundary. So allow for sb with source_sad above threshold,
+ // and avoid very large source_sad or high source content, to avoid
+ // too many 8x8 within superblock.
+ if (segment_id == 0 && cpi->rc.avg_source_sad < 25000 &&
+ blk_sad > 25000 && blk_sad < 50000 && !lighting_change) {
+ thresholds[2] = (3 * thresholds[2]) >> 2;
+ thresholds[3] = thresholds[2] << 3;
+ }
// Condition the increase of partition thresholds on the segment
// and the content. Avoid the increase for superblocks which have
// high source sad, unless the whole frame has very high motion
@@ -602,7 +615,7 @@
(source_sad_nonrd != kHighSad ||
cpi->rc.avg_source_sad > 50000)) {
thresholds[0] = (3 * thresholds[0]) >> 1;
- thresholds[3] = INT32_MAX;
+ thresholds[3] = INT64_MAX;
if (current_qindex > QINDEX_LARGE_BLOCK_THR) {
thresholds[1] =
(int)((1 - weight) * (thresholds[1] << 1) + weight * thresholds[1]);
@@ -616,16 +629,16 @@
(int)((1 - weight) * (thresholds[1] << 2) + weight * thresholds[1]);
thresholds[2] =
(int)((1 - weight) * (thresholds[2] << 4) + weight * thresholds[2]);
- thresholds[3] = INT32_MAX;
+ thresholds[3] = INT64_MAX;
}
} else if (cpi->sf.rt_sf.prefer_large_partition_blocks >= 2) {
- tune_thresh_based_on_qindex_window(
- current_qindex, QINDEX_LARGE_BLOCK_THR, source_sad_nonrd,
- cpi->sf.rt_sf.prefer_large_partition_blocks - 2, thresholds);
+ thresholds[1] <<= (source_sad_nonrd <= kLowSad) ? 2 : 0;
+ thresholds[2] =
+ (source_sad_nonrd <= kLowSad) ? (3 * thresholds[2]) : thresholds[2];
} else if (cpi->sf.rt_sf.prefer_large_partition_blocks >= 1) {
- thresholds[3] <<= 2;
- thresholds[1] <<= (source_sad_nonrd == kLowSad) ? 1 : 0;
- thresholds[2] <<= (source_sad_nonrd == kLowSad) ? 1 : 0;
+ const int fac = (source_sad_nonrd <= kLowSad) ? 2 : 1;
+ tune_thresh_based_on_qindex_window(current_qindex, QINDEX_LARGE_BLOCK_THR,
+ 45, fac, thresholds);
}
if (cpi->sf.part_sf.disable_8x8_part_based_on_qidx && (current_qindex < 128))
thresholds[3] = INT64_MAX;
@@ -916,7 +929,7 @@
return;
} else {
set_vbp_thresholds(cpi, cpi->vbp_info.thresholds, q, content_lowsumdiff, 0,
- 0, 0);
+ 0, 0, 0, 0);
// The threshold below is not changed locally.
cpi->vbp_info.threshold_minmax = 15 + (q >> 3);
}
@@ -1010,13 +1023,23 @@
const int compute_minmax_variance = 0;
const int segment_id = xd->mi[0]->segment_id;
int pixels_wide = 128, pixels_high = 128;
-
+ int border_offset_4x4 = 0;
+ int temporal_denoising = cpi->sf.rt_sf.use_rtc_tf;
if (is_small_sb) {
pixels_wide = 64;
pixels_high = 64;
}
if (xd->mb_to_right_edge < 0) pixels_wide += (xd->mb_to_right_edge >> 3);
if (xd->mb_to_bottom_edge < 0) pixels_high += (xd->mb_to_bottom_edge >> 3);
+#if CONFIG_AV1_TEMPORAL_DENOISING
+ temporal_denoising |= cpi->oxcf.noise_sensitivity;
+#endif
+ // For temporal filtering or temporal denoiser enabled: since the source
+ // is modified we need to avoid 4x4 avg along superblock boundary, since
+ // simd code will load 8 pixels for 4x4 avg and so can access source
+ // data outside superblock (while its being modified by temporal filter).
+ // Temporal filtering is never done on key frames.
+ if (!is_key_frame && temporal_denoising) border_offset_4x4 = 4;
for (int m = 0; m < num_64x64_blocks; m++) {
const int x64_idx = ((m & 1) << 6);
const int y64_idx = ((m >> 1) << 6);
@@ -1096,12 +1119,12 @@
int x8_idx = x16_idx + ((k & 1) << 3);
int y8_idx = y16_idx + ((k >> 1) << 3);
VP8x8 *vst2 = is_key_frame ? &vst->split[k] : &vt2[i2 + j].split[k];
- fill_variance_4x4avg(src, src_stride, dst, dst_stride, x8_idx,
- y8_idx, vst2,
+ fill_variance_4x4avg(
+ src, src_stride, dst, dst_stride, x8_idx, y8_idx, vst2,
#if CONFIG_AV1_HIGHBITDEPTH
- xd->cur_buf->flags,
+ xd->cur_buf->flags,
#endif
- pixels_wide, pixels_high, is_key_frame);
+ pixels_wide, pixels_high, is_key_frame, border_offset_4x4);
}
}
}
@@ -1205,6 +1228,18 @@
: PART_EVAL_ONLY_NONE;
}
+static AOM_INLINE bool is_set_force_zeromv_skip_based_on_src_sad(
+ int set_zeromv_skip_based_on_source_sad, SOURCE_SAD source_sad_nonrd) {
+ if (set_zeromv_skip_based_on_source_sad == 0) return false;
+
+ if (set_zeromv_skip_based_on_source_sad >= 2)
+ return source_sad_nonrd <= kVeryLowSad;
+ else if (set_zeromv_skip_based_on_source_sad >= 1)
+ return source_sad_nonrd == kZeroSad;
+
+ return false;
+}
+
int av1_choose_var_based_partitioning(AV1_COMP *cpi, const TileInfo *const tile,
ThreadData *td, MACROBLOCK *x, int mi_row,
int mi_col) {
@@ -1267,6 +1302,17 @@
const int low_res = (cm->width <= 352 && cm->height <= 288);
int variance4x4downsample[64];
const int segment_id = xd->mi[0]->segment_id;
+ uint64_t blk_sad = 0;
+ if (cpi->src_sad_blk_64x64 != NULL) {
+ const int sb_size_by_mb = (cm->seq_params->sb_size == BLOCK_128X128)
+ ? (cm->seq_params->mib_size >> 1)
+ : cm->seq_params->mib_size;
+ const int sb_cols =
+ (cm->mi_params.mi_cols + sb_size_by_mb - 1) / sb_size_by_mb;
+ const int sbi_col = mi_col / sb_size_by_mb;
+ const int sbi_row = mi_row / sb_size_by_mb;
+ blk_sad = cpi->src_sad_blk_64x64[sbi_col + sbi_row * sb_cols];
+ }
if (cpi->oxcf.q_cfg.aq_mode == CYCLIC_REFRESH_AQ && cm->seg.enabled &&
cyclic_refresh_segment_id_boosted(segment_id)) {
@@ -1274,12 +1320,14 @@
av1_get_qindex(&cm->seg, segment_id, cm->quant_params.base_qindex);
set_vbp_thresholds(cpi, thresholds, q, x->content_state_sb.low_sumdiff,
x->content_state_sb.source_sad_nonrd,
- x->content_state_sb.source_sad_rd, 1);
+ x->content_state_sb.source_sad_rd, 1, blk_sad,
+ x->content_state_sb.lighting_change);
} else {
set_vbp_thresholds(cpi, thresholds, cm->quant_params.base_qindex,
x->content_state_sb.low_sumdiff,
x->content_state_sb.source_sad_nonrd,
- x->content_state_sb.source_sad_rd, 0);
+ x->content_state_sb.source_sad_rd, 0, blk_sad,
+ x->content_state_sb.lighting_change);
}
// For non keyframes, disable 4x4 average for low resolution when speed = 8
@@ -1333,32 +1381,40 @@
chroma_check(cpi, x, bsize, y_sad_last, y_sad_g, is_key_frame, zero_motion,
uv_sad);
- x->force_zeromv_skip = 0;
- const unsigned int thresh_exit_part =
- (cm->seq_params->sb_size == BLOCK_64X64) ? 5000 : 10000;
+ x->force_zeromv_skip_for_sb = 0;
+ const bool is_set_force_zeromv_skip =
+ is_set_force_zeromv_skip_based_on_src_sad(
+ cpi->sf.rt_sf.set_zeromv_skip_based_on_source_sad,
+ x->content_state_sb.source_sad_nonrd);
+
// If the superblock is completely static (zero source sad) and
// the y_sad (relative to LAST ref) is very small, take the sb_size partition
// and exit, and force zeromv_last skip mode for nonrd_pickmode.
- // Only do this when the cyclic refresh is applied, and only on the base
- // segment (so the QP-boosted segment can still contnue cleaning/ramping
- // up the quality). Condition on color uv_sad is also added.
+ // Only do this on the base segment (so the QP-boosted segment, if applied,
+ // can still continue cleaning/ramping up the quality).
+ // Condition on color uv_sad is also added.
if (!is_key_frame && cpi->sf.rt_sf.part_early_exit_zeromv &&
- cpi->oxcf.q_cfg.aq_mode == CYCLIC_REFRESH_AQ &&
- cpi->cyclic_refresh->apply_cyclic_refresh &&
- segment_id == CR_SEGMENT_ID_BASE &&
- x->content_state_sb.source_sad_nonrd == kZeroSad &&
- ref_frame_partition == LAST_FRAME && xd->mi[0]->mv[0].as_int == 0 &&
- y_sad < thresh_exit_part && uv_sad[0]<(3 * thresh_exit_part)>> 2 &&
- uv_sad[1]<(3 * thresh_exit_part)>> 2) {
+ cpi->rc.frames_since_key > 30 && segment_id == CR_SEGMENT_ID_BASE &&
+ is_set_force_zeromv_skip && ref_frame_partition == LAST_FRAME &&
+ xd->mi[0]->mv[0].as_int == 0) {
const int block_width = mi_size_wide[cm->seq_params->sb_size];
const int block_height = mi_size_high[cm->seq_params->sb_size];
+ const unsigned int thresh_exit_part_y =
+ cpi->zeromv_skip_thresh_exit_part[bsize];
+ const unsigned int thresh_exit_part_uv =
+ CALC_CHROMA_THRESH_FOR_ZEROMV_SKIP(thresh_exit_part_y);
if (mi_col + block_width <= tile->mi_col_end &&
- mi_row + block_height <= tile->mi_row_end) {
+ mi_row + block_height <= tile->mi_row_end &&
+ y_sad < thresh_exit_part_y && uv_sad[0] < thresh_exit_part_uv &&
+ uv_sad[1] < thresh_exit_part_uv) {
set_block_size(cpi, mi_row, mi_col, bsize);
- x->force_zeromv_skip = 1;
+ x->force_zeromv_skip_for_sb = 1;
if (vt2) aom_free(vt2);
if (vt) aom_free(vt);
return 0;
+ } else if (x->content_state_sb.source_sad_nonrd == kZeroSad &&
+ cpi->sf.rt_sf.part_early_exit_zeromv >= 2) {
+ x->force_zeromv_skip_for_sb = 2;
}
}
diff --git a/av1/encoder/var_based_part.h b/av1/encoder/var_based_part.h
index 5176751..7febc0e 100644
--- a/av1/encoder/var_based_part.h
+++ b/av1/encoder/var_based_part.h
@@ -28,6 +28,8 @@
100 // Use increased thresholds for midres for speed 9 when qindex is above
// this threshold
+#define CALC_CHROMA_THRESH_FOR_ZEROMV_SKIP(thresh_exit_part) \
+ ((3 * (thresh_exit_part)) >> 2)
/*!\brief Set the thresholds for variance based partition.
*
* Set the variance split thresholds for following the block sizes:
@@ -44,7 +46,7 @@
* \param[in] q q index
* \param[in] content_lowsumdiff Low sumdiff flag for superblock
*
- * \return Returns the set of thresholds in \c cpi->vbp_info.thresholds.
+ * \remark Returns the set of thresholds in \c cpi->vbp_info.thresholds.
*/
void av1_set_variance_partition_thresholds(AV1_COMP *cpi, int q,
int content_lowsumdiff);
diff --git a/av1/encoder/x86/av1_fwd_txfm_sse2.c b/av1/encoder/x86/av1_fwd_txfm_sse2.c
index 694e613..a5cbe16 100644
--- a/av1/encoder/x86/av1_fwd_txfm_sse2.c
+++ b/av1/encoder/x86/av1_fwd_txfm_sse2.c
@@ -1425,7 +1425,7 @@
const __m128i sinpi_p03_p04 = pair_set_epi16(sinpi[3], sinpi[4]);
const __m128i sinpi_m03_p02 = pair_set_epi16(-sinpi[3], sinpi[2]);
const __m128i sinpi_p03_p03 = _mm_set1_epi16((int16_t)sinpi[3]);
- const __m128i __zero = _mm_set1_epi16(0);
+ const __m128i __zero = _mm_setzero_si128();
const __m128i __rounding = _mm_set1_epi32(1 << (cos_bit - 1));
const __m128i in7 = _mm_add_epi16(input[0], input[1]);
__m128i u[8], v[8];
@@ -1573,7 +1573,7 @@
const __m128i sinpi_p03_p04 = pair_set_epi16(sinpi[3], sinpi[4]);
const __m128i sinpi_m03_p02 = pair_set_epi16(-sinpi[3], sinpi[2]);
const __m128i sinpi_p03_p03 = _mm_set1_epi16((int16_t)sinpi[3]);
- const __m128i __zero = _mm_set1_epi16(0);
+ const __m128i __zero = _mm_setzero_si128();
const __m128i __rounding = _mm_set1_epi32(1 << (cos_bit - 1));
const __m128i in7 = _mm_add_epi16(input[0], input[1]);
__m128i u_lo[8], u_hi[8], v_lo[8], v_hi[8];
diff --git a/av1/encoder/x86/av1_k_means_avx2.c b/av1/encoder/x86/av1_k_means_avx2.c
index 23a7369..759f515 100644
--- a/av1/encoder/x86/av1_k_means_avx2.c
+++ b/av1/encoder/x86/av1_k_means_avx2.c
@@ -21,7 +21,7 @@
for (int i = 0; i < n; i += 8) {
__m256i ind = _mm256_loadu_si256((__m256i *)data);
for (int j = 0; j < k; j++) {
- __m256i cent = _mm256_set1_epi32((uint32_t)centroids[j]);
+ __m256i cent = _mm256_set1_epi32(centroids[j]);
__m256i d1 = _mm256_sub_epi32(ind, cent);
dist[j] = _mm256_mullo_epi32(d1, d1);
}
diff --git a/av1/encoder/x86/av1_k_means_sse2.c b/av1/encoder/x86/av1_k_means_sse2.c
index 43f661f..f03c459 100644
--- a/av1/encoder/x86/av1_k_means_sse2.c
+++ b/av1/encoder/x86/av1_k_means_sse2.c
@@ -25,7 +25,7 @@
l = (l == 0) ? 1 : 0;
ind[l] = _mm_loadu_si128((__m128i *)data);
for (int j = 0; j < k; j++) {
- __m128i cent = _mm_set1_epi32((uint32_t)centroids[j]);
+ __m128i cent = _mm_set1_epi32(centroids[j]);
__m128i d1 = _mm_sub_epi32(ind[l], cent);
__m128i d2 = _mm_packs_epi32(d1, d1);
__m128i d3 = _mm_mullo_epi16(d2, d2);
diff --git a/av1/encoder/x86/highbd_block_error_intrin_sse2.c b/av1/encoder/x86/highbd_block_error_intrin_sse2.c
index 4579e4e..0287f01 100644
--- a/av1/encoder/x86/highbd_block_error_intrin_sse2.c
+++ b/av1/encoder/x86/highbd_block_error_intrin_sse2.c
@@ -33,7 +33,7 @@
__m128i mm_dqcoeff2 = _mm_load_si128((__m128i *)(dqcoeff + i + 4));
// Check if any values require more than 15 bit
max = _mm_set1_epi32(0x3fff);
- min = _mm_set1_epi32(0xffffc000);
+ min = _mm_set1_epi32((int)0xffffc000);
cmp0 = _mm_xor_si128(_mm_cmpgt_epi32(mm_coeff, max),
_mm_cmplt_epi32(mm_coeff, min));
cmp1 = _mm_xor_si128(_mm_cmpgt_epi32(mm_coeff2, max),
diff --git a/av1/encoder/x86/pickrst_avx2.c b/av1/encoder/x86/pickrst_avx2.c
index d53b128..e244d5e 100644
--- a/av1/encoder/x86/pickrst_avx2.c
+++ b/av1/encoder/x86/pickrst_avx2.c
@@ -59,7 +59,7 @@
M_int[k][l] += D1 * X1 + D2 * X2;
const __m256i kl =
- _mm256_cvtepu8_epi16(_mm_set1_epi16(loadu_uint16(dgd_ijk + l)));
+ _mm256_cvtepu8_epi16(_mm_set1_epi16(loadu_int16(dgd_ijk + l)));
acc_stat_avx2(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle, &kl);
acc_stat_avx2(H_ + 1 * 8, dgd_ij + 1 * dgd_stride, shuffle, &kl);
acc_stat_avx2(H_ + 2 * 8, dgd_ij + 2 * dgd_stride, shuffle, &kl);
@@ -88,7 +88,7 @@
// are (effectively) used as inputs to a multiply-accumulate.
// So if we set the extra pixel slot to 0, then it is effectively
// ignored.
- const __m256i kl = _mm256_cvtepu8_epi16(_mm_set1_epi16((uint16_t)D1));
+ const __m256i kl = _mm256_cvtepu8_epi16(_mm_set1_epi16((int16_t)D1));
acc_stat_avx2(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle, &kl);
acc_stat_avx2(H_ + 1 * 8, dgd_ij + 1 * dgd_stride, shuffle, &kl);
acc_stat_avx2(H_ + 2 * 8, dgd_ij + 2 * dgd_stride, shuffle, &kl);
@@ -260,7 +260,7 @@
// Load two u16 values from dgd_ijkl combined as a u32,
// then broadcast to 8x u32 slots of a 256
- const __m256i dgd_ijkl = _mm256_set1_epi32(loadu_uint32(dgd_ijk + l));
+ const __m256i dgd_ijkl = _mm256_set1_epi32(loadu_int32(dgd_ijk + l));
// dgd_ijkl = [y x y x y x y x] [y x y x y x y x] where each is a u16
acc_stat_highbd_avx2(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle,
@@ -297,7 +297,7 @@
// interleaved copies of two pixels, but we only have one. However, the
// pixels are (effectively) used as inputs to a multiply-accumulate. So
// if we set the extra pixel slot to 0, then it is effectively ignored.
- const __m256i dgd_ijkl = _mm256_set1_epi32((uint32_t)D1);
+ const __m256i dgd_ijkl = _mm256_set1_epi32((int)D1);
acc_stat_highbd_avx2(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle,
&dgd_ijkl);
@@ -408,7 +408,7 @@
// Load two u16 values from dgd_ijkl combined as a u32,
// then broadcast to 8x u32 slots of a 256
- const __m256i dgd_ijkl = _mm256_set1_epi32(loadu_uint32(dgd_ijk + l));
+ const __m256i dgd_ijkl = _mm256_set1_epi32(loadu_int32(dgd_ijk + l));
// dgd_ijkl = [x y x y x y x y] [x y x y x y x y] where each is a u16
acc_stat_highbd_avx2(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle,
@@ -441,7 +441,7 @@
// interleaved copies of two pixels, but we only have one. However, the
// pixels are (effectively) used as inputs to a multiply-accumulate. So
// if we set the extra pixel slot to 0, then it is effectively ignored.
- const __m256i dgd_ijkl = _mm256_set1_epi32((uint32_t)D1);
+ const __m256i dgd_ijkl = _mm256_set1_epi32((int)D1);
acc_stat_highbd_avx2(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle,
&dgd_ijkl);
@@ -569,7 +569,7 @@
M_int[k][l] += D1 * X1 + D2 * X2;
const __m256i kl =
- _mm256_cvtepu8_epi16(_mm_set1_epi16(loadu_uint16(dgd_ijk + l)));
+ _mm256_cvtepu8_epi16(_mm_set1_epi16(loadu_int16(dgd_ijk + l)));
acc_stat_avx2(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle, &kl);
acc_stat_avx2(H_ + 1 * 8, dgd_ij + 1 * dgd_stride, shuffle, &kl);
acc_stat_avx2(H_ + 2 * 8, dgd_ij + 2 * dgd_stride, shuffle, &kl);
@@ -596,7 +596,7 @@
// are (effectively) used as inputs to a multiply-accumulate.
// So if we set the extra pixel slot to 0, then it is effectively
// ignored.
- const __m256i kl = _mm256_cvtepu8_epi16(_mm_set1_epi16((uint16_t)D1));
+ const __m256i kl = _mm256_cvtepu8_epi16(_mm_set1_epi16((int16_t)D1));
acc_stat_avx2(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle, &kl);
acc_stat_avx2(H_ + 1 * 8, dgd_ij + 1 * dgd_stride, shuffle, &kl);
acc_stat_avx2(H_ + 2 * 8, dgd_ij + 2 * dgd_stride, shuffle, &kl);
diff --git a/av1/encoder/x86/pickrst_sse4.c b/av1/encoder/x86/pickrst_sse4.c
index 3d496ef..8208cca 100644
--- a/av1/encoder/x86/pickrst_sse4.c
+++ b/av1/encoder/x86/pickrst_sse4.c
@@ -62,7 +62,7 @@
M_int[k][l] += D1 * X1 + D2 * X2;
const __m128i kl =
- _mm_cvtepu8_epi16(_mm_set1_epi16(*((uint16_t *)(dgd_ijk + l))));
+ _mm_cvtepu8_epi16(_mm_set1_epi16(*((int16_t *)(dgd_ijk + l))));
acc_stat_sse41(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle, &kl);
acc_stat_sse41(H_ + 1 * 8, dgd_ij + 1 * dgd_stride, shuffle, &kl);
acc_stat_sse41(H_ + 2 * 8, dgd_ij + 2 * dgd_stride, shuffle, &kl);
@@ -91,7 +91,7 @@
// are (effectively) used as inputs to a multiply-accumulate.
// So if we set the extra pixel slot to 0, then it is effectively
// ignored.
- const __m128i kl = _mm_cvtepu8_epi16(_mm_set1_epi16((uint16_t)D1));
+ const __m128i kl = _mm_cvtepu8_epi16(_mm_set1_epi16((int16_t)D1));
acc_stat_sse41(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle, &kl);
acc_stat_sse41(H_ + 1 * 8, dgd_ij + 1 * dgd_stride, shuffle, &kl);
acc_stat_sse41(H_ + 2 * 8, dgd_ij + 2 * dgd_stride, shuffle, &kl);
@@ -265,7 +265,7 @@
// Load two u16 values from dgd as a single u32
// Then broadcast to 4x u32 slots of a 128
- const __m128i dgd_ijkl = _mm_set1_epi32(*((uint32_t *)(dgd_ijk + l)));
+ const __m128i dgd_ijkl = _mm_set1_epi32(*((int *)(dgd_ijk + l)));
// dgd_ijkl = [y x y x y x y x] as u16
acc_stat_highbd_sse41(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle,
@@ -302,7 +302,7 @@
// interleaved copies of two pixels, but we only have one. However, the
// pixels are (effectively) used as inputs to a multiply-accumulate. So
// if we set the extra pixel slot to 0, then it is effectively ignored.
- const __m128i dgd_ijkl = _mm_set1_epi32((uint32_t)D1);
+ const __m128i dgd_ijkl = _mm_set1_epi32((int)D1);
acc_stat_highbd_sse41(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle,
&dgd_ijkl);
@@ -414,7 +414,7 @@
// Load two u16 values from dgd as a single u32
// then broadcast to 4x u32 slots of a 128
- const __m128i dgd_ijkl = _mm_set1_epi32(*((uint32_t *)(dgd_ijk + l)));
+ const __m128i dgd_ijkl = _mm_set1_epi32(*((int *)(dgd_ijk + l)));
// dgd_ijkl = [y x y x y x y x] as u16
acc_stat_highbd_sse41(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle,
@@ -447,7 +447,7 @@
// interleaved copies of two pixels, but we only have one. However, the
// pixels are (effectively) used as inputs to a multiply-accumulate. So
// if we set the extra pixel slot to 0, then it is effectively ignored.
- const __m128i dgd_ijkl = _mm_set1_epi32((uint32_t)D1);
+ const __m128i dgd_ijkl = _mm_set1_epi32((int)D1);
acc_stat_highbd_sse41(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle,
&dgd_ijkl);
@@ -574,7 +574,7 @@
M_int[k][l] += D1 * X1 + D2 * X2;
const __m128i kl =
- _mm_cvtepu8_epi16(_mm_set1_epi16(*((uint16_t *)(dgd_ijk + l))));
+ _mm_cvtepu8_epi16(_mm_set1_epi16(*((int16_t *)(dgd_ijk + l))));
acc_stat_sse41(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle, &kl);
acc_stat_sse41(H_ + 1 * 8, dgd_ij + 1 * dgd_stride, shuffle, &kl);
acc_stat_sse41(H_ + 2 * 8, dgd_ij + 2 * dgd_stride, shuffle, &kl);
@@ -601,7 +601,7 @@
// are (effectively) used as inputs to a multiply-accumulate.
// So if we set the extra pixel slot to 0, then it is effectively
// ignored.
- const __m128i kl = _mm_cvtepu8_epi16(_mm_set1_epi16((uint16_t)D1));
+ const __m128i kl = _mm_cvtepu8_epi16(_mm_set1_epi16((int16_t)D1));
acc_stat_sse41(H_ + 0 * 8, dgd_ij + 0 * dgd_stride, shuffle, &kl);
acc_stat_sse41(H_ + 1 * 8, dgd_ij + 1 * dgd_stride, shuffle, &kl);
acc_stat_sse41(H_ + 2 * 8, dgd_ij + 2 * dgd_stride, shuffle, &kl);
diff --git a/av1/encoder/x86/rdopt_avx2.c b/av1/encoder/x86/rdopt_avx2.c
index 3bc763c..a0ab394 100644
--- a/av1/encoder/x86/rdopt_avx2.c
+++ b/av1/encoder/x86/rdopt_avx2.c
@@ -31,8 +31,8 @@
// [ m n o p ]
const __m256i pixels = _mm256_set_epi64x(
- loadu_uint64(&diff[0 * stride]), loadu_uint64(&diff[1 * stride]),
- loadu_uint64(&diff[2 * stride]), loadu_uint64(&diff[3 * stride]));
+ loadu_int64(&diff[0 * stride]), loadu_int64(&diff[1 * stride]),
+ loadu_int64(&diff[2 * stride]), loadu_int64(&diff[3 * stride]));
// pixels = [d c b a h g f e] [l k j i p o n m] as i16
const __m256i slli = _mm256_slli_epi64(pixels, 16);
diff --git a/av1/encoder/x86/rdopt_sse4.c b/av1/encoder/x86/rdopt_sse4.c
index 4c4ec1f..12ac146 100644
--- a/av1/encoder/x86/rdopt_sse4.c
+++ b/av1/encoder/x86/rdopt_sse4.c
@@ -29,10 +29,10 @@
// [ i j k l ]
// [ m n o p ]
- const __m128i pixelsa = _mm_set_epi64x(*(uint64_t *)&diff[0 * stride],
- *(uint64_t *)&diff[2 * stride]);
- const __m128i pixelsb = _mm_set_epi64x(*(uint64_t *)&diff[1 * stride],
- *(uint64_t *)&diff[3 * stride]);
+ const __m128i pixelsa = _mm_set_epi64x(*(int64_t *)&diff[0 * stride],
+ *(int64_t *)&diff[2 * stride]);
+ const __m128i pixelsb = _mm_set_epi64x(*(int64_t *)&diff[1 * stride],
+ *(int64_t *)&diff[3 * stride]);
// pixelsa = [d c b a l k j i] as i16
// pixelsb = [h g f e p o n m] as i16
diff --git a/av1/encoder/x86/reconinter_enc_sse2.c b/av1/encoder/x86/reconinter_enc_sse2.c
index 6455bf3..d33fec7 100644
--- a/av1/encoder/x86/reconinter_enc_sse2.c
+++ b/av1/encoder/x86/reconinter_enc_sse2.c
@@ -305,13 +305,12 @@
assert(!(width * height & 7));
n = width * height >> 3;
- const uint16_t wt0 = (uint16_t)jcp_param->fwd_offset;
- const uint16_t wt1 = (uint16_t)jcp_param->bck_offset;
- const __m128i w0 = _mm_set_epi16(wt0, wt0, wt0, wt0, wt0, wt0, wt0, wt0);
- const __m128i w1 = _mm_set_epi16(wt1, wt1, wt1, wt1, wt1, wt1, wt1, wt1);
- const uint16_t round = ((1 << DIST_PRECISION_BITS) >> 1);
- const __m128i r =
- _mm_set_epi16(round, round, round, round, round, round, round, round);
+ const int16_t wt0 = (int16_t)jcp_param->fwd_offset;
+ const int16_t wt1 = (int16_t)jcp_param->bck_offset;
+ const __m128i w0 = _mm_set1_epi16(wt0);
+ const __m128i w1 = _mm_set1_epi16(wt1);
+ const int16_t round = (int16_t)((1 << DIST_PRECISION_BITS) >> 1);
+ const __m128i r = _mm_set1_epi16(round);
uint16_t *comp_pred16 = CONVERT_TO_SHORTPTR(comp_pred8);
for (i = 0; i < n; i++) {
diff --git a/av1/encoder/x86/reconinter_enc_ssse3.c b/av1/encoder/x86/reconinter_enc_ssse3.c
index 7ac0f0d..df7aa95 100644
--- a/av1/encoder/x86/reconinter_enc_ssse3.c
+++ b/av1/encoder/x86/reconinter_enc_ssse3.c
@@ -48,13 +48,12 @@
assert(!(width * height & 15));
n = width * height >> 4;
- const uint8_t w0 = (uint8_t)jcp_param->fwd_offset;
- const uint8_t w1 = (uint8_t)jcp_param->bck_offset;
+ const int8_t w0 = (int8_t)jcp_param->fwd_offset;
+ const int8_t w1 = (int8_t)jcp_param->bck_offset;
const __m128i w = _mm_set_epi8(w1, w0, w1, w0, w1, w0, w1, w0, w1, w0, w1, w0,
w1, w0, w1, w0);
- const uint16_t round = ((1 << DIST_PRECISION_BITS) >> 1);
- const __m128i r =
- _mm_set_epi16(round, round, round, round, round, round, round, round);
+ const int16_t round = (int16_t)((1 << DIST_PRECISION_BITS) >> 1);
+ const __m128i r = _mm_set1_epi16(round);
for (i = 0; i < n; i++) {
__m128i p0 = xx_loadu_128(comp_pred);
diff --git a/av1/encoder/x86/wedge_utils_avx2.c b/av1/encoder/x86/wedge_utils_avx2.c
index c06bad8..bbc62d5 100644
--- a/av1/encoder/x86/wedge_utils_avx2.c
+++ b/av1/encoder/x86/wedge_utils_avx2.c
@@ -31,7 +31,7 @@
uint64_t csse;
const __m256i v_mask_max_w = _mm256_set1_epi16(MAX_MASK_VALUE);
- const __m256i v_zext_q = yy_set1_64_from_32i(0xffffffff);
+ const __m256i v_zext_q = yy_set1_64_from_32i(~0);
__m256i v_acc0_q = _mm256_setzero_si256();
@@ -142,7 +142,7 @@
v_acc_q_0 = _mm_add_epi64(v_acc_q_0, v_acc_q_1);
#if ARCH_X86_64
- acc = (uint64_t)_mm_extract_epi64(v_acc_q_0, 0);
+ acc = _mm_extract_epi64(v_acc_q_0, 0);
#else
xx_storel_64(&acc, v_acc_q_0);
#endif
@@ -155,7 +155,7 @@
*/
void av1_wedge_compute_delta_squares_avx2(int16_t *d, const int16_t *a,
const int16_t *b, int N) {
- const __m256i v_neg_w = _mm256_set1_epi32(0xffff0001);
+ const __m256i v_neg_w = _mm256_set1_epi32((int)0xffff0001);
assert(N % 64 == 0);
diff --git a/av1/encoder/x86/wedge_utils_sse2.c b/av1/encoder/x86/wedge_utils_sse2.c
index f3f4b8a..e665b2e 100644
--- a/av1/encoder/x86/wedge_utils_sse2.c
+++ b/av1/encoder/x86/wedge_utils_sse2.c
@@ -31,7 +31,7 @@
uint64_t csse;
const __m128i v_mask_max_w = _mm_set1_epi16(MAX_MASK_VALUE);
- const __m128i v_zext_q = xx_set1_64_from_32i(0xffffffff);
+ const __m128i v_zext_q = xx_set1_64_from_32i(~0);
__m128i v_acc0_q = _mm_setzero_si128();
@@ -175,7 +175,7 @@
v_acc_q = _mm_add_epi64(v_acc_q, _mm_srli_si128(v_acc_q, 8));
#if ARCH_X86_64
- acc = (uint64_t)_mm_cvtsi128_si64(v_acc_q);
+ acc = _mm_cvtsi128_si64(v_acc_q);
#else
xx_storel_64(&acc, v_acc_q);
#endif
diff --git a/av1/ratectrl_qmode.cc b/av1/ratectrl_qmode.cc
index 8edefd7..e02c21e 100644
--- a/av1/ratectrl_qmode.cc
+++ b/av1/ratectrl_qmode.cc
@@ -68,7 +68,7 @@
gop_frame->is_key_frame = 0;
gop_frame->is_arf_frame = 1;
gop_frame->is_show_frame = 0;
- gop_frame->is_golden_frame = 0;
+ gop_frame->is_golden_frame = gop_frame->layer_depth <= 2 ? 1 : 0;
gop_frame->encode_ref_mode = EncodeRefMode::kRegular;
break;
case GopFrameType::kRegularLeaf:
@@ -106,10 +106,10 @@
gop_frame.display_idx = display_idx;
gop_frame.global_coding_idx = global_coding_idx_offset + coding_idx;
gop_frame.global_order_idx = global_order_idx_offset + order_idx;
- SetGopFrameByType(gop_frame_type, &gop_frame);
+ gop_frame.layer_depth = depth + kLayerDepthOffset;
gop_frame.colocated_ref_idx = -1;
gop_frame.update_ref_idx = -1;
- gop_frame.layer_depth = depth + kLayerDepthOffset;
+ SetGopFrameByType(gop_frame_type, &gop_frame);
return gop_frame;
}
@@ -201,13 +201,13 @@
ref_frame_manager->UpdateRefFrameTable(&gop_frame);
gop_struct.gop_frame_list.push_back(gop_frame);
ConstructGopMultiLayer(&gop_struct, ref_frame_manager,
- ref_frame_manager->ForwardMaxSize(), arf_depth + 1,
+ ref_frame_manager->MaxRefFrame() - 1, arf_depth + 1,
order_start, order_end);
// Overlay
gop_frame =
GopFrameBasic(global_coding_idx_offset, global_order_idx_offset,
static_cast<int>(gop_struct.gop_frame_list.size()),
- order_end, ref_frame_manager->ForwardMaxSize(),
+ order_end, ref_frame_manager->MaxRefFrame() - 1,
gop_struct.display_tracker, GopFrameType::kOverlay);
ref_frame_manager->UpdateRefFrameTable(&gop_frame);
gop_struct.gop_frame_list.push_back(gop_frame);
@@ -249,11 +249,6 @@
<< ") must be in the range [1, 7].";
return { AOM_CODEC_INVALID_PARAM, error_message.str() };
}
- if (rc_param.max_depth < 1 || rc_param.max_depth > 5) {
- error_message << "max_depth (" << rc_param.max_depth
- << ") must be in the range [1, 5].";
- return { AOM_CODEC_INVALID_PARAM, error_message.str() };
- }
if (rc_param.base_q_index < 0 || rc_param.base_q_index > 255) {
error_message << "base_q_index (" << rc_param.base_q_index
<< ") must be in the range [0, 255].";
@@ -854,7 +849,8 @@
const FirstpassInfo &firstpass_info) {
const int stats_size = static_cast<int>(firstpass_info.stats_list.size());
GopStructList gop_list;
- RefFrameManager ref_frame_manager(rc_param_.ref_frame_table_size);
+ RefFrameManager ref_frame_manager(rc_param_.ref_frame_table_size,
+ rc_param_.max_ref_frames);
int global_coding_idx_offset = 0;
int global_order_idx_offset = 0;
@@ -1153,7 +1149,9 @@
}
std::vector<RefFrameTable> AV1RateControlQMode::GetRefFrameTableList(
- const GopStruct &gop_struct, RefFrameTable ref_frame_table) {
+ const GopStruct &gop_struct,
+ const std::vector<LookaheadStats> &lookahead_stats,
+ RefFrameTable ref_frame_table) {
if (gop_struct.global_coding_idx_offset == 0) {
// For the first GOP, ref_frame_table need not be initialized. This is fine,
// because the first frame (a key frame) will fully initialize it.
@@ -1180,14 +1178,46 @@
}
ref_frame_table_list.push_back(ref_frame_table);
}
+
+ int gop_size_offset = static_cast<int>(gop_struct.gop_frame_list.size());
+
+ for (const auto &lookahead_stat : lookahead_stats) {
+ for (GopFrame gop_frame : lookahead_stat.gop_struct->gop_frame_list) {
+ if (gop_frame.is_key_frame) {
+ ref_frame_table.assign(rc_param_.ref_frame_table_size, gop_frame);
+ } else if (gop_frame.update_ref_idx != -1) {
+ assert(gop_frame.update_ref_idx <
+ static_cast<int>(ref_frame_table.size()));
+ gop_frame.coding_idx += gop_size_offset;
+ ref_frame_table[gop_frame.update_ref_idx] = gop_frame;
+ }
+ ref_frame_table_list.push_back(ref_frame_table);
+ }
+ gop_size_offset +=
+ static_cast<int>(lookahead_stat.gop_struct->gop_frame_list.size());
+ }
+
return ref_frame_table_list;
}
StatusOr<TplGopDepStats> ComputeTplGopDepStats(
const TplGopStats &tpl_gop_stats,
+ const std::vector<LookaheadStats> &lookahead_stats,
const std::vector<RefFrameTable> &ref_frame_table_list) {
+ std::vector<const TplFrameStats *> tpl_frame_stats_list_with_lookahead;
+ for (const auto &tpl_frame_stats : tpl_gop_stats.frame_stats_list) {
+ tpl_frame_stats_list_with_lookahead.push_back(&tpl_frame_stats);
+ }
+ for (auto &lookahead_stat : lookahead_stats) {
+ for (const auto &tpl_frame_stats :
+ lookahead_stat.tpl_gop_stats->frame_stats_list) {
+ tpl_frame_stats_list_with_lookahead.push_back(&tpl_frame_stats);
+ }
+ }
+
const int frame_count =
- static_cast<int>(tpl_gop_stats.frame_stats_list.size());
+ static_cast<int>(tpl_frame_stats_list_with_lookahead.size());
+
// Create the struct to store TPL dependency stats
TplGopDepStats tpl_gop_dep_stats;
@@ -1195,7 +1225,7 @@
for (int coding_idx = 0; coding_idx < frame_count; coding_idx++) {
const StatusOr<TplFrameDepStats> tpl_frame_dep_stats =
CreateTplFrameDepStatsWithoutPropagation(
- tpl_gop_stats.frame_stats_list[coding_idx]);
+ *tpl_frame_stats_list_with_lookahead[coding_idx]);
if (!tpl_frame_dep_stats.ok()) {
return tpl_frame_dep_stats.status();
}
@@ -1231,19 +1261,28 @@
StatusOr<GopEncodeInfo> AV1RateControlQMode::GetGopEncodeInfo(
const GopStruct &gop_struct, const TplGopStats &tpl_gop_stats,
+ const std::vector<LookaheadStats> &lookahead_stats,
const RefFrameTable &ref_frame_table_snapshot_init) {
Status status = ValidateTplStats(gop_struct, tpl_gop_stats);
if (!status.ok()) {
return status;
}
- const std::vector<RefFrameTable> ref_frame_table_list =
- GetRefFrameTableList(gop_struct, ref_frame_table_snapshot_init);
+ for (auto &lookahead_stat : lookahead_stats) {
+ Status status = ValidateTplStats(*lookahead_stat.gop_struct,
+ *lookahead_stat.tpl_gop_stats);
+ if (!status.ok()) {
+ return status;
+ }
+ }
+
+ const std::vector<RefFrameTable> ref_frame_table_list = GetRefFrameTableList(
+ gop_struct, lookahead_stats, ref_frame_table_snapshot_init);
GopEncodeInfo gop_encode_info;
gop_encode_info.final_snapshot = ref_frame_table_list.back();
- StatusOr<TplGopDepStats> gop_dep_stats =
- ComputeTplGopDepStats(tpl_gop_stats, ref_frame_table_list);
+ StatusOr<TplGopDepStats> gop_dep_stats = ComputeTplGopDepStats(
+ tpl_gop_stats, lookahead_stats, ref_frame_table_list);
if (!gop_dep_stats.ok()) {
return gop_dep_stats.status();
}
diff --git a/av1/ratectrl_qmode.h b/av1/ratectrl_qmode.h
index 7a59687..5f7dd86 100644
--- a/av1/ratectrl_qmode.h
+++ b/av1/ratectrl_qmode.h
@@ -87,6 +87,7 @@
StatusOr<TplGopDepStats> ComputeTplGopDepStats(
const TplGopStats &tpl_gop_stats,
+ const std::vector<LookaheadStats> &lookahead_stats,
const std::vector<RefFrameTable> &ref_frame_table_list);
class AV1RateControlQMode : public AV1RateControlQModeInterface {
@@ -96,6 +97,7 @@
const FirstpassInfo &firstpass_info) override;
StatusOr<GopEncodeInfo> GetGopEncodeInfo(
const GopStruct &gop_struct, const TplGopStats &tpl_gop_stats,
+ const std::vector<LookaheadStats> &lookahead_stats,
const RefFrameTable &ref_frame_table_snapshot) override;
// Public for testing only.
@@ -104,7 +106,9 @@
// If this is first GOP, ref_frame_table is ignored and all refs are assumed
// invalid; otherwise ref_frame_table is used as the initial state.
std::vector<RefFrameTable> GetRefFrameTableList(
- const GopStruct &gop_struct, RefFrameTable ref_frame_table);
+ const GopStruct &gop_struct,
+ const std::vector<LookaheadStats> &lookahead_stats,
+ RefFrameTable ref_frame_table);
private:
RateControlParam rc_param_;
diff --git a/av1/ratectrl_qmode_interface.h b/av1/ratectrl_qmode_interface.h
index 1e4e5c6..9dbbcbb 100644
--- a/av1/ratectrl_qmode_interface.h
+++ b/av1/ratectrl_qmode_interface.h
@@ -24,8 +24,10 @@
constexpr int kBlockRefCount = 2;
struct MotionVector {
- int row; // subpel row
- int col; // subpel col
+ int row; // subpel row
+ int col; // subpel col
+ // TODO(b/241589513): Move this to TPLFrameStats; it's wasteful to code it
+ // separately for each block.
int subpel_bits; // number of fractional bits used by row/col
};
@@ -37,9 +39,6 @@
int ref_frame_table_size;
// Maximum number of references a single frame may use.
int max_ref_frames;
- // Maximum pyramid depth. e.g., 1 means only one ARF per GOP,
- // 2 would allow an additional level of intermediate ARFs.
- int max_depth;
int base_q_index;
@@ -48,12 +47,17 @@
};
struct TplBlockStats {
- int height; // pixel height
- int width; // pixel width
- int row; // pixel row of the top left corner
- int col; // pixel col of the top lef corner
+ int16_t height; // Pixel height.
+ int16_t width; // Pixel width.
+ int16_t row; // Pixel row of the top left corner.
+ int16_t col; // Pixel col of the top lef corner.
int64_t intra_cost;
int64_t inter_cost;
+
+ // Valid only if TplFrameStats::rate_dist_present is true:
+ int64_t recrf_rate; // Bits when using recon as reference.
+ int64_t recrf_dist; // Distortion when using recon as reference.
+
std::array<MotionVector, kBlockRefCount> mv;
std::array<int, kBlockRefCount> ref_frame_index;
};
@@ -233,6 +237,7 @@
int min_block_size;
int frame_width;
int frame_height;
+ bool rate_dist_present; // True if recrf_rate and recrf_dist are populated.
std::vector<TplBlockStats> block_stats_list;
};
@@ -240,6 +245,12 @@
std::vector<TplFrameStats> frame_stats_list;
};
+// Structure and TPL stats for a single GOP, to be used for lookahead.
+struct LookaheadStats {
+ const GopStruct *gop_struct; // Not owned, may not be nullptr.
+ const TplGopStats *tpl_gop_stats; // Not owned, may not be nullptr.
+};
+
class AV1RateControlQModeInterface {
public:
AV1RateControlQModeInterface();
@@ -248,14 +259,21 @@
virtual Status SetRcParam(const RateControlParam &rc_param) = 0;
virtual StatusOr<GopStructList> DetermineGopInfo(
const FirstpassInfo &firstpass_info) = 0;
- // Accept firstpass and TPL info from the encoder and return q index and
- // rdmult. This needs to be called with consecutive GOPs as returned by
- // DetermineGopInfo.
+
+ // Accepts GOP structure and TPL info from the encoder and returns q index and
+ // rdmult for each frame. This should be called with consecutive GOPs as
+ // returned by DetermineGopInfo.
+ //
+ // GOP structure and TPL info from zero or more subsequent GOPs may optionally
+ // be passed in lookahead_stats.
+ //
// For the first GOP, a default-constructed RefFrameTable may be passed in as
// ref_frame_table_snapshot_init; for subsequent GOPs, it should be the
// final_snapshot returned on the previous call.
+ //
virtual StatusOr<GopEncodeInfo> GetGopEncodeInfo(
const GopStruct &gop_struct, const TplGopStats &tpl_gop_stats,
+ const std::vector<LookaheadStats> &lookahead_stats,
const RefFrameTable &ref_frame_table_snapshot_init) = 0;
};
} // namespace aom
diff --git a/av1/ratectrl_rtc.cc b/av1/ratectrl_rtc.cc
index f1af797..db6fbcd 100644
--- a/av1/ratectrl_rtc.cc
+++ b/av1/ratectrl_rtc.cc
@@ -65,8 +65,11 @@
rc_api->cpi_->ppi =
static_cast<AV1_PRIMARY *>(aom_memalign(32, sizeof(AV1_PRIMARY)));
if (!rc_api->cpi_->ppi) return nullptr;
+ av1_zero(*rc_api->cpi_->ppi);
rc_api->cpi_->common.seq_params = &rc_api->cpi_->ppi->seq_params;
av1_zero(*rc_api->cpi_->common.seq_params);
+ const int num_layers = cfg.ss_number_layers * cfg.ts_number_layers;
+ av1_alloc_layer_context(rc_api->cpi_, num_layers);
rc_api->InitRateControl(cfg);
if (cfg.aq_mode) {
AV1_COMP *const cpi = rc_api->cpi_;
@@ -94,6 +97,9 @@
}
}
}
+ aom_free(cpi_->svc.layer_context);
+ cpi_->svc.layer_context = nullptr;
+
if (cpi_->oxcf.q_cfg.aq_mode == CYCLIC_REFRESH_AQ) {
aom_free(cpi_->enc_seg.map);
cpi_->enc_seg.map = nullptr;
diff --git a/av1/reference_manager.cc b/av1/reference_manager.cc
index 456cec9..032a0b8 100644
--- a/av1/reference_manager.cc
+++ b/av1/reference_manager.cc
@@ -80,11 +80,18 @@
int priority_idx) const {
if (ref_update_type == RefUpdateType::kForward) {
int size = static_cast<int>(forward_stack_.size());
+ // When two or more forward reference frames can be used, first get
+ // the highest quality one as the ARF, then going from nearest to
+ // the more distant ones in the forward reference frame list.
if (priority_idx < size) {
- if (priority_idx == 0)
- return forward_stack_[priority_idx];
- else
+ if (allow_two_fwd_frames_) {
+ if (priority_idx == 0) return forward_stack_[0];
return forward_stack_[size - priority_idx];
+ }
+
+ // Handle the special case where only one forward reference frame
+ // can be used. In this setting, we prefer the nearest frame.
+ return forward_stack_[size - 1 - priority_idx];
}
} else if (ref_update_type == RefUpdateType::kBackward) {
int size = static_cast<int>(backward_queue_.size());
@@ -126,8 +133,8 @@
const std::vector<ReferenceName> forward_name_list{
ReferenceName::kAltrefFrame, ReferenceName::kBwdrefFrame,
ReferenceName::kAltref2Frame, ReferenceName::kGoldenFrame,
- ReferenceName::kLastFrame, ReferenceName::kLast2Frame,
- ReferenceName::kLast3Frame
+ ReferenceName::kLast3Frame, ReferenceName::kLast2Frame,
+ ReferenceName::kLastFrame
};
const std::vector<ReferenceName> backward_name_list{
ReferenceName::kGoldenFrame, ReferenceName::kLastFrame,
@@ -178,8 +185,10 @@
std::vector<ReferenceFrame> ref_frame_list;
int ref_frame_count = 0;
int round_robin_idx = 0;
+
std::set<ReferenceName> used_name_set;
- while (ref_frame_count < available_ref_frames) {
+ while (ref_frame_count < available_ref_frames &&
+ ref_frame_count < max_ref_frames_) {
const RefUpdateType ref_update_type = round_robin_list[round_robin_idx];
int priority_idx = priority_idx_list[round_robin_idx];
int ref_idx = GetRefFrameIdxByPriority(ref_update_type, priority_idx);
@@ -270,13 +279,14 @@
const GopFrame &gop_frame) const {
assert(gop_frame.is_valid);
std::vector<std::pair<PrimaryRefKey, int>> candidate_list;
- for (int ref_idx = 0; ref_idx < static_cast<int>(ref_frame_table_.size());
- ++ref_idx) {
- const GopFrame &ref_frame = ref_frame_table_[ref_idx];
+ for (auto &ref_frame_in_gop_frame : gop_frame.ref_frame_list) {
+ const GopFrame &ref_frame = ref_frame_table_[ref_frame_in_gop_frame.index];
if (ref_frame.is_valid) {
- assert(ref_idx == ref_frame.update_ref_idx);
+ assert(ref_frame_in_gop_frame.index == ref_frame.update_ref_idx);
PrimaryRefKey key = get_primary_ref_key(gop_frame, ref_frame);
- std::pair<PrimaryRefKey, int> candidate = { key, ref_idx };
+ std::pair<PrimaryRefKey, int> candidate = {
+ key, ref_frame_in_gop_frame.index
+ };
candidate_list.push_back(candidate);
}
}
@@ -284,11 +294,10 @@
std::sort(candidate_list.begin(), candidate_list.end());
ReferenceFrame ref_frame = { -1, ReferenceName::kNoneFrame };
- std::vector<ReferenceFrame> ref_frame_list = GetRefFrameListByPriority();
- assert(candidate_list.size() == ref_frame_list.size());
+ assert(candidate_list.size() == gop_frame.ref_frame_list.size());
if (!candidate_list.empty()) {
int ref_idx = candidate_list[0].second;
- for (const auto &frame : ref_frame_list) {
+ for (const auto &frame : gop_frame.ref_frame_list) {
if (frame.index == ref_idx) {
ref_frame = frame;
}
@@ -298,6 +307,9 @@
}
void RefFrameManager::UpdateRefFrameTable(GopFrame *gop_frame) {
+ allow_two_fwd_frames_ =
+ (max_ref_frames_ - !!GetRefFrameCountByType(RefUpdateType::kBackward) -
+ !!GetRefFrameCountByType(RefUpdateType::kLast)) >= 2;
gop_frame->ref_frame_list = GetRefFrameListByPriority();
gop_frame->primary_ref_frame = GetPrimaryRefFrame(*gop_frame);
gop_frame->colocated_ref_idx = ColocatedRefIdx(gop_frame->global_order_idx);
diff --git a/av1/reference_manager.h b/av1/reference_manager.h
index 59bfda3..951d65d 100644
--- a/av1/reference_manager.h
+++ b/av1/reference_manager.h
@@ -24,8 +24,9 @@
class RefFrameManager {
public:
- explicit RefFrameManager(int ref_frame_table_size)
- : ref_frame_table_(ref_frame_table_size) {
+ explicit RefFrameManager(int ref_frame_table_size, int max_ref_frames)
+ : ref_frame_table_(ref_frame_table_size),
+ max_ref_frames_(max_ref_frames) {
// forward_max_size_ define max number of arf frames that can exists at
// the same time. In the other words, it's the max size of forward_stack_.
// TODO(angiebird): Figure out if this number is optimal.
@@ -72,6 +73,7 @@
void UpdateOrder(int global_order_idx);
int ColocatedRefIdx(int global_order_idx);
int ForwardMaxSize() const { return forward_max_size_; }
+ int MaxRefFrame() const { return max_ref_frames_; }
int CurGlobalOrderIdx() const { return cur_global_order_idx_; }
void UpdateRefFrameTable(GopFrame *gop_frame);
ReferenceFrame GetPrimaryRefFrame(const GopFrame &gop_frame) const;
@@ -80,6 +82,8 @@
int forward_max_size_;
int cur_global_order_idx_;
RefFrameTable ref_frame_table_;
+ int max_ref_frames_;
+ bool allow_two_fwd_frames_;
std::deque<int> free_ref_idx_list_;
std::vector<int> forward_stack_;
std::deque<int> backward_queue_;
diff --git a/build/cmake/aom_config_defaults.cmake b/build/cmake/aom_config_defaults.cmake
index 84159fb..b8e68f8 100644
--- a/build/cmake/aom_config_defaults.cmake
+++ b/build/cmake/aom_config_defaults.cmake
@@ -31,6 +31,7 @@
# ARM feature flags.
set_aom_detect_var(HAVE_NEON 0 "Enables NEON intrinsics optimizations.")
+set_aom_detect_var(HAVE_ARM_CRC32 0 "Enables Arm CRC32 optimizations.")
# MIPS feature flags.
set_aom_detect_var(HAVE_DSPR2 0 "Enables DSPR2 optimizations.")
diff --git a/build/cmake/aom_configure.cmake b/build/cmake/aom_configure.cmake
index 26c521e..c00888f 100644
--- a/build/cmake/aom_configure.cmake
+++ b/build/cmake/aom_configure.cmake
@@ -40,6 +40,10 @@
"FORCE_HIGHBITDEPTH_DECODING")
endif()
+if(CONFIG_THREE_PASS AND NOT CONFIG_AV1_DECODER)
+ change_config_and_warn(CONFIG_THREE_PASS 0 "CONFIG_AV1_DECODER=0")
+endif()
+
# Generate the user config settings.
list(APPEND aom_build_vars ${AOM_CONFIG_VARS} ${AOM_OPTION_VARS})
foreach(cache_var ${aom_build_vars})
@@ -300,7 +304,17 @@
add_compiler_flag_if_supported("-Wall")
add_compiler_flag_if_supported("-Wdisabled-optimization")
add_compiler_flag_if_supported("-Wextra")
- add_compiler_flag_if_supported("-Wextra-semi")
+ # Prior to version 3.19.0 cmake would fail to parse the warning emitted by gcc
+ # with this flag. Note the order of this check and -Wextra-semi-stmt is
+ # important due to is_flag_present() matching substrings with string(FIND
+ # ...).
+ if(CMAKE_VERSION VERSION_LESS "3.19"
+ AND CMAKE_C_COMPILER_ID STREQUAL "GNU"
+ AND CMAKE_C_COMPILER_VERSION VERSION_GREATER_EQUAL 10)
+ add_cxx_flag_if_supported("-Wextra-semi")
+ else()
+ add_compiler_flag_if_supported("-Wextra-semi")
+ endif()
add_compiler_flag_if_supported("-Wextra-semi-stmt")
add_compiler_flag_if_supported("-Wfloat-conversion")
add_compiler_flag_if_supported("-Wformat=2")
@@ -314,6 +328,7 @@
add_compiler_flag_if_supported("-Wuninitialized")
add_compiler_flag_if_supported("-Wunused")
add_compiler_flag_if_supported("-Wvla")
+ add_cxx_flag_if_supported("-Wc++14-extensions")
add_cxx_flag_if_supported("-Wc++17-extensions")
add_cxx_flag_if_supported("-Wc++20-extensions")
diff --git a/build/cmake/cpu.cmake b/build/cmake/cpu.cmake
index ef2d755..c818ec6 100644
--- a/build/cmake/cpu.cmake
+++ b/build/cmake/cpu.cmake
@@ -20,6 +20,19 @@
set(HAVE_NEON 0)
set(AOM_RTCD_FLAGS ${AOM_RTCD_FLAGS} --disable-neon)
endif()
+
+ check_c_source_compiles("
+ #if !defined(__ARM_FEATURE_CRC32) || __ARM_FEATURE_CRC32 != 1
+ #error \"CRC32 is unavailable.\"
+ #endif
+ int main(void) { return 0; }" HAVE_CRC32)
+ if(HAVE_CRC32)
+ set(HAVE_ARM_CRC32 1)
+ else()
+ set(HAVE_ARM_CRC32 0)
+ set(AOM_RTCD_FLAGS ${AOM_RTCD_FLAGS} --disable-arm_crc32)
+ endif()
+
elseif("${AOM_TARGET_CPU}" MATCHES "^mips")
set(ARCH_MIPS 1)
set(RTCD_ARCH_MIPS "yes")
diff --git a/build/cmake/rtcd.pl b/build/cmake/rtcd.pl
index e9f75dd..7f961ca 100755
--- a/build/cmake/rtcd.pl
+++ b/build/cmake/rtcd.pl
@@ -432,8 +432,8 @@
@ALL_ARCHS = filter(qw/neon/);
arm;
} elsif ($opts{arch} eq 'arm64' ) {
- @ALL_ARCHS = filter(qw/neon/);
- &require("neon");
+ @ALL_ARCHS = filter(qw/neon arm_crc32/);
+ &require(@ALL_ARCHS);
arm;
} elsif ($opts{arch} eq 'ppc') {
@ALL_ARCHS = filter(qw/vsx/);
diff --git a/build/cmake/toolchains/arm64-linux-gcc.cmake b/build/cmake/toolchains/arm64-linux-gcc.cmake
index fc4b277..64e460b 100644
--- a/build/cmake/toolchains/arm64-linux-gcc.cmake
+++ b/build/cmake/toolchains/arm64-linux-gcc.cmake
@@ -21,9 +21,15 @@
set(CROSS aarch64-linux-gnu-)
endif()
-set(CMAKE_C_COMPILER ${CROSS}gcc)
-set(CMAKE_CXX_COMPILER ${CROSS}g++)
-set(AS_EXECUTABLE ${CROSS}as)
+if(NOT CMAKE_C_COMPILER)
+ set(CMAKE_C_COMPILER ${CROSS}gcc)
+endif()
+if(NOT CMAKE_CXX_COMPILER)
+ set(CMAKE_CXX_COMPILER ${CROSS}g++)
+endif()
+if(NOT AS_EXECUTABLE)
+ set(AS_EXECUTABLE ${CROSS}as)
+endif()
set(CMAKE_C_FLAGS_INIT "-march=armv8-a")
set(CMAKE_CXX_FLAGS_INIT "-march=armv8-a")
set(AOM_AS_FLAGS "-march=armv8-a")
diff --git a/build/cmake/toolchains/arm64-mingw-gcc.cmake b/build/cmake/toolchains/arm64-mingw-gcc.cmake
index a8e15cb..5472ed4 100644
--- a/build/cmake/toolchains/arm64-mingw-gcc.cmake
+++ b/build/cmake/toolchains/arm64-mingw-gcc.cmake
@@ -20,10 +20,18 @@
set(CROSS aarch64-w64-mingw32-)
endif()
-set(CMAKE_C_COMPILER ${CROSS}gcc)
-set(CMAKE_CXX_COMPILER ${CROSS}g++)
-set(CMAKE_AR ${CROSS}ar CACHE FILEPATH Archiver)
-set(CMAKE_RANLIB ${CROSS}ranlib CACHE FILEPATH Indexer)
+if(NOT CMAKE_C_COMPILER)
+ set(CMAKE_C_COMPILER ${CROSS}gcc)
+endif()
+if(NOT CMAKE_CXX_COMPILER)
+ set(CMAKE_CXX_COMPILER ${CROSS}g++)
+endif()
+if(NOT CMAKE_AR)
+ set(CMAKE_AR ${CROSS}ar CACHE FILEPATH Archiver)
+endif()
+if(NOT CMAKE_RANLIB)
+ set(CMAKE_RANLIB ${CROSS}ranlib CACHE FILEPATH Indexer)
+endif()
# No runtime cpu detect for arm64-mingw-gcc.
set(CONFIG_RUNTIME_CPU_DETECT 0 CACHE STRING "")
diff --git a/build/cmake/toolchains/armv7-linux-gcc.cmake b/build/cmake/toolchains/armv7-linux-gcc.cmake
index 26c028f..1201538 100644
--- a/build/cmake/toolchains/armv7-linux-gcc.cmake
+++ b/build/cmake/toolchains/armv7-linux-gcc.cmake
@@ -25,9 +25,15 @@
set(AOM_EXTRA_TOOLCHAIN_FLAGS "-mfloat-abi=softfp")
endif()
-set(CMAKE_C_COMPILER ${CROSS}gcc)
-set(CMAKE_CXX_COMPILER ${CROSS}g++)
-set(AS_EXECUTABLE ${CROSS}as)
+if(NOT CMAKE_C_COMPILER)
+ set(CMAKE_C_COMPILER ${CROSS}gcc)
+endif()
+if(NOT CMAKE_CXX_COMPILER)
+ set(CMAKE_CXX_COMPILER ${CROSS}g++)
+endif()
+if(NOT AS_EXECUTABLE)
+ set(AS_EXECUTABLE ${CROSS}as)
+endif()
set(CMAKE_C_FLAGS_INIT "-march=armv7-a -mfpu=vfpv3 \
${AOM_EXTRA_TOOLCHAIN_FLAGS}")
set(CMAKE_CXX_FLAGS_INIT "-march=armv7-a -mfpu=vfpv3 \
diff --git a/build/cmake/toolchains/armv7-mingw-gcc.cmake b/build/cmake/toolchains/armv7-mingw-gcc.cmake
index 2dc4b18..8a92891 100644
--- a/build/cmake/toolchains/armv7-mingw-gcc.cmake
+++ b/build/cmake/toolchains/armv7-mingw-gcc.cmake
@@ -20,10 +20,18 @@
set(CROSS armv7-w64-mingw32-)
endif()
-set(CMAKE_C_COMPILER ${CROSS}gcc)
-set(CMAKE_CXX_COMPILER ${CROSS}g++)
-set(CMAKE_AR ${CROSS}ar CACHE FILEPATH Archiver)
-set(CMAKE_RANLIB ${CROSS}ranlib CACHE FILEPATH Indexer)
+if(NOT CMAKE_C_COMPILER)
+ set(CMAKE_C_COMPILER ${CROSS}gcc)
+endif()
+if(NOT CMAKE_CXX_COMPILER)
+ set(CMAKE_CXX_COMPILER ${CROSS}g++)
+endif()
+if(NOT CMAKE_AR)
+ set(CMAKE_AR ${CROSS}ar CACHE FILEPATH Archiver)
+endif()
+if(NOT CMAKE_RANLIB)
+ set(CMAKE_RANLIB ${CROSS}ranlib CACHE FILEPATH Indexer)
+endif()
# No runtime cpu detect for armv7-mingw-gcc.
set(CONFIG_RUNTIME_CPU_DETECT 0 CACHE STRING "")
diff --git a/build/cmake/toolchains/mips32-linux-gcc.cmake b/build/cmake/toolchains/mips32-linux-gcc.cmake
index ad5ebff..15bed17 100644
--- a/build/cmake/toolchains/mips32-linux-gcc.cmake
+++ b/build/cmake/toolchains/mips32-linux-gcc.cmake
@@ -62,9 +62,15 @@
set(CMAKE_EXE_LINKER_FLAGS "-mfp64 ${CMAKE_EXE_LINKER_FLAGS}")
endif()
-set(CMAKE_C_COMPILER ${CROSS}gcc)
-set(CMAKE_CXX_COMPILER ${CROSS}g++)
-set(AS_EXECUTABLE ${CROSS}as)
+if(NOT CMAKE_C_COMPILER)
+ set(CMAKE_C_COMPILER ${CROSS}gcc)
+endif()
+if(NOT CMAKE_CXX_COMPILER)
+ set(CMAKE_CXX_COMPILER ${CROSS}g++)
+endif()
+if(NOT AS_EXECUTABLE)
+ set(AS_EXECUTABLE ${CROSS}as)
+endif()
set(CMAKE_C_FLAGS_INIT "-EL ${MIPS_CFLAGS}")
set(CMAKE_CXX_FLAGS_INIT "-EL ${MIPS_CXXFLAGS}")
set(CMAKE_EXE_LINKER_FLAGS_INIT "-EL ${MIPS_CXXFLAGS}")
diff --git a/build/cmake/toolchains/mips64-linux-gcc.cmake b/build/cmake/toolchains/mips64-linux-gcc.cmake
index 0af99245..93e7d22 100644
--- a/build/cmake/toolchains/mips64-linux-gcc.cmake
+++ b/build/cmake/toolchains/mips64-linux-gcc.cmake
@@ -39,9 +39,15 @@
"-mips64r6 -mabi64 -mfp64 ${CMAKE_EXE_LINKER_FLAGS}")
endif()
-set(CMAKE_C_COMPILER ${CROSS}gcc)
-set(CMAKE_CXX_COMPILER ${CROSS}g++)
-set(AS_EXECUTABLE ${CROSS}as)
+if(NOT CMAKE_C_COMPILER)
+ set(CMAKE_C_COMPILER ${CROSS}gcc)
+endif()
+if(NOT CMAKE_CXX_COMPILER)
+ set(CMAKE_CXX_COMPILER ${CROSS}g++)
+endif()
+if(NOT AS_EXECUTABLE)
+ set(AS_EXECUTABLE ${CROSS}as)
+endif()
set(CMAKE_C_FLAGS_INIT "-EL ${MIPS_CFLAGS}")
set(CMAKE_CXX_FLAGS_INIT "-EL ${MIPS_CXXFLAGS}")
set(CMAKE_EXE_LINKER_FLAGS_INIT "-EL ${MIPS_CXXFLAGS}")
diff --git a/build/cmake/toolchains/ppc-linux-gcc.cmake b/build/cmake/toolchains/ppc-linux-gcc.cmake
index 54db99b..ab0efea 100644
--- a/build/cmake/toolchains/ppc-linux-gcc.cmake
+++ b/build/cmake/toolchains/ppc-linux-gcc.cmake
@@ -21,9 +21,15 @@
set(CROSS powerpc64le-unknown-linux-gnu-)
endif()
-set(CMAKE_C_COMPILER ${CROSS}gcc)
-set(CMAKE_CXX_COMPILER ${CROSS}g++)
-set(AS_EXECUTABLE ${CROSS}as)
+if(NOT CMAKE_C_COMPILER)
+ set(CMAKE_C_COMPILER ${CROSS}gcc)
+endif()
+if(NOT CMAKE_CXX_COMPILER)
+ set(CMAKE_CXX_COMPILER ${CROSS}g++)
+endif()
+if(NOT AS_EXECUTABLE)
+ set(AS_EXECUTABLE ${CROSS}as)
+endif()
set(CMAKE_SYSTEM_PROCESSOR "ppc")
set(CONFIG_RUNTIME_CPU_DETECT 0 CACHE STRING "")
diff --git a/build/cmake/toolchains/x86-mingw-gcc.cmake b/build/cmake/toolchains/x86-mingw-gcc.cmake
index 2e9a9a8..f75728f 100644
--- a/build/cmake/toolchains/x86-mingw-gcc.cmake
+++ b/build/cmake/toolchains/x86-mingw-gcc.cmake
@@ -23,7 +23,15 @@
set(CROSS i686-w64-mingw32-)
endif()
-set(CMAKE_C_COMPILER ${CROSS}gcc)
-set(CMAKE_CXX_COMPILER ${CROSS}g++)
-set(CMAKE_AR ${CROSS}ar CACHE FILEPATH Archiver)
-set(CMAKE_RANLIB ${CROSS}ranlib CACHE FILEPATH Indexer)
+if(NOT CMAKE_C_COMPILER)
+ set(CMAKE_C_COMPILER ${CROSS}gcc)
+endif()
+if(NOT CMAKE_CXX_COMPILER)
+ set(CMAKE_CXX_COMPILER ${CROSS}g++)
+endif()
+if(NOT CMAKE_AR)
+ set(CMAKE_AR ${CROSS}ar CACHE FILEPATH Archiver)
+endif()
+if(NOT CMAKE_RANLIB)
+ set(CMAKE_RANLIB ${CROSS}ranlib CACHE FILEPATH Indexer)
+endif()
diff --git a/build/cmake/toolchains/x86_64-mingw-gcc.cmake b/build/cmake/toolchains/x86_64-mingw-gcc.cmake
index 4b2d28d..56e9b6e 100644
--- a/build/cmake/toolchains/x86_64-mingw-gcc.cmake
+++ b/build/cmake/toolchains/x86_64-mingw-gcc.cmake
@@ -20,7 +20,15 @@
set(CROSS x86_64-w64-mingw32-)
endif()
-set(CMAKE_C_COMPILER ${CROSS}gcc)
-set(CMAKE_CXX_COMPILER ${CROSS}g++)
-set(CMAKE_AR ${CROSS}ar CACHE FILEPATH Archiver)
-set(CMAKE_RANLIB ${CROSS}ranlib CACHE FILEPATH Indexer)
+if(NOT CMAKE_C_COMPILER)
+ set(CMAKE_C_COMPILER ${CROSS}gcc)
+endif()
+if(NOT CMAKE_CXX_COMPILER)
+ set(CMAKE_CXX_COMPILER ${CROSS}g++)
+endif()
+if(NOT CMAKE_AR)
+ set(CMAKE_AR ${CROSS}ar CACHE FILEPATH Archiver)
+endif()
+if(NOT CMAKE_RANLIB)
+ set(CMAKE_RANLIB ${CROSS}ranlib CACHE FILEPATH Indexer)
+endif()
diff --git a/common/md5_utils.c b/common/md5_utils.c
index b69e1cc..c69aa57 100644
--- a/common/md5_utils.c
+++ b/common/md5_utils.c
@@ -150,19 +150,26 @@
#define AOM_NO_UNSIGNED_OVERFLOW_CHECK \
__attribute__((no_sanitize("unsigned-integer-overflow")))
#endif
-#endif
+#if __clang_major__ >= 12
+#define VPX_NO_UNSIGNED_SHIFT_CHECK \
+ __attribute__((no_sanitize("unsigned-shift-base")))
+#endif // __clang__ >= 12
+#endif // __clang__
#ifndef AOM_NO_UNSIGNED_OVERFLOW_CHECK
#define AOM_NO_UNSIGNED_OVERFLOW_CHECK
#endif
+#ifndef AOM_NO_UNSIGNED_SHIFT_CHECK
+#define AOM_NO_UNSIGNED_SHIFT_CHECK
+#endif
/*
* The core of the MD5 algorithm, this alters an existing MD5 hash to
* reflect the addition of 16 longwords of new data. MD5Update blocks
* the data and converts bytes into longwords for this routine.
*/
-AOM_NO_UNSIGNED_OVERFLOW_CHECK void MD5Transform(UWORD32 buf[4],
- UWORD32 const in[16]) {
+AOM_NO_UNSIGNED_OVERFLOW_CHECK AOM_NO_UNSIGNED_SHIFT_CHECK void MD5Transform(
+ UWORD32 buf[4], UWORD32 const in[16]) {
register UWORD32 a, b, c, d;
a = buf[0];
@@ -245,5 +252,6 @@
}
#undef AOM_NO_UNSIGNED_OVERFLOW_CHECK
+#undef AOM_NO_UNSIGNED_SHIFT_CHECK
#endif
diff --git a/common/obudec.c b/common/obudec.c
index 650f997..a71a6dd 100644
--- a/common/obudec.c
+++ b/common/obudec.c
@@ -288,6 +288,7 @@
if (obudec_read_leb128(f, &detect_buf[0], &length_of_unit_size,
&unit_size) != 0) {
fprintf(stderr, "obudec: Failure reading temporal unit header\n");
+ rewind(f);
return 0;
}
@@ -295,6 +296,7 @@
if (obudec_read_leb128(f, &detect_buf[length_of_unit_size],
&annexb_header_length, &unit_size) != 0) {
fprintf(stderr, "obudec: Failure reading frame unit header\n");
+ rewind(f);
return 0;
}
annexb_header_length += length_of_unit_size;
@@ -316,6 +318,7 @@
if (obu_header.type != OBU_TEMPORAL_DELIMITER &&
obu_header.type != OBU_SEQUENCE_HEADER) {
+ rewind(f);
return 0;
}
@@ -350,6 +353,7 @@
if (payload_length > (obu_ctx->buffer_capacity - bytes_read)) {
fprintf(stderr, "obudec: First OBU's payload is too large\n");
rewind(f);
+ obudec_free(obu_ctx);
return 0;
}
@@ -358,6 +362,7 @@
f, payload_length, &obu_ctx->buffer[bytes_read], &payload_bytes);
if (status < 0) {
rewind(f);
+ obudec_free(obu_ctx);
return 0;
}
obu_ctx->bytes_buffered += payload_bytes;
@@ -483,4 +488,9 @@
return 0;
}
-void obudec_free(struct ObuDecInputContext *obu_ctx) { free(obu_ctx->buffer); }
+void obudec_free(struct ObuDecInputContext *obu_ctx) {
+ free(obu_ctx->buffer);
+ obu_ctx->buffer = NULL;
+ obu_ctx->buffer_capacity = 0;
+ obu_ctx->bytes_buffered = 0;
+}
diff --git a/common/webmdec.h b/common/webmdec.h
index 5ac75cb..fcbdeff 100644
--- a/common/webmdec.h
+++ b/common/webmdec.h
@@ -28,7 +28,7 @@
const void *block;
int block_frame_index;
int video_track_index;
- uint64_t timestamp_ns;
+ int64_t timestamp_ns;
int is_key_frame;
int reached_eos;
};
diff --git a/common/y4menc.c b/common/y4menc.c
index eaeedba..7d32465 100644
--- a/common/y4menc.c
+++ b/common/y4menc.c
@@ -52,30 +52,25 @@
switch (bit_depth) {
case 8: return colorspace8(csp, fmt);
case 9:
- return fmt == AOM_IMG_FMT_I44416
- ? "C444p9 XYSCSS=444P9"
- : fmt == AOM_IMG_FMT_I42216 ? "C422p9 XYSCSS=422P9"
- : "C420p9 XYSCSS=420P9";
+ return fmt == AOM_IMG_FMT_I44416 ? "C444p9 XYSCSS=444P9"
+ : fmt == AOM_IMG_FMT_I42216 ? "C422p9 XYSCSS=422P9"
+ : "C420p9 XYSCSS=420P9";
case 10:
- return fmt == AOM_IMG_FMT_I44416
- ? "C444p10 XYSCSS=444P10"
- : fmt == AOM_IMG_FMT_I42216 ? "C422p10 XYSCSS=422P10"
- : "C420p10 XYSCSS=420P10";
+ return fmt == AOM_IMG_FMT_I44416 ? "C444p10 XYSCSS=444P10"
+ : fmt == AOM_IMG_FMT_I42216 ? "C422p10 XYSCSS=422P10"
+ : "C420p10 XYSCSS=420P10";
case 12:
- return fmt == AOM_IMG_FMT_I44416
- ? "C444p12 XYSCSS=444P12"
- : fmt == AOM_IMG_FMT_I42216 ? "C422p12 XYSCSS=422P12"
- : "C420p12 XYSCSS=420P12";
+ return fmt == AOM_IMG_FMT_I44416 ? "C444p12 XYSCSS=444P12"
+ : fmt == AOM_IMG_FMT_I42216 ? "C422p12 XYSCSS=422P12"
+ : "C420p12 XYSCSS=420P12";
case 14:
- return fmt == AOM_IMG_FMT_I44416
- ? "C444p14 XYSCSS=444P14"
- : fmt == AOM_IMG_FMT_I42216 ? "C422p14 XYSCSS=422P14"
- : "C420p14 XYSCSS=420P14";
+ return fmt == AOM_IMG_FMT_I44416 ? "C444p14 XYSCSS=444P14"
+ : fmt == AOM_IMG_FMT_I42216 ? "C422p14 XYSCSS=422P14"
+ : "C420p14 XYSCSS=420P14";
case 16:
- return fmt == AOM_IMG_FMT_I44416
- ? "C444p16 XYSCSS=444P16"
- : fmt == AOM_IMG_FMT_I42216 ? "C422p16 XYSCSS=422P16"
- : "C420p16 XYSCSS=420P16";
+ return fmt == AOM_IMG_FMT_I44416 ? "C444p16 XYSCSS=444P16"
+ : fmt == AOM_IMG_FMT_I42216 ? "C422p16 XYSCSS=422P16"
+ : "C420p16 XYSCSS=420P16";
default: assert(0); return NULL;
}
}
diff --git a/examples/svc_encoder_rtc.c b/examples/svc_encoder_rtc.c
index 7629a1b..e68ff3c 100644
--- a/examples/svc_encoder_rtc.c
+++ b/examples/svc_encoder_rtc.c
@@ -37,6 +37,7 @@
int aq_mode;
int layering_mode;
int output_obu;
+ int decode;
} AppInput;
typedef enum {
@@ -87,6 +88,9 @@
static const arg_def_t output_obu_arg =
ARG_DEF(NULL, "output-obu", 1,
"Write OBUs when set to 1. Otherwise write IVF files.");
+static const arg_def_t test_decode_arg =
+ ARG_DEF(NULL, "test-decode", 1,
+ "Attempt to test decoding the output when set to 1. Default is 1.");
#if CONFIG_AV1_HIGHBITDEPTH
static const struct arg_enum_list bitdepth_enum[] = {
@@ -97,18 +101,31 @@
"d", "bit-depth", 1, "Bit depth for codec 8, 10 or 12. ", bitdepth_enum);
#endif // CONFIG_AV1_HIGHBITDEPTH
-static const arg_def_t *svc_args[] = {
- &frames_arg, &outputfile, &width_arg,
- &height_arg, &timebase_arg, &bitrate_arg,
- &spatial_layers_arg, &kf_dist_arg, &scale_factors_arg,
- &min_q_arg, &max_q_arg, &temporal_layers_arg,
- &layering_mode_arg, &threads_arg, &aqmode_arg,
+static const arg_def_t *svc_args[] = { &frames_arg,
+ &outputfile,
+ &width_arg,
+ &height_arg,
+ &timebase_arg,
+ &bitrate_arg,
+ &spatial_layers_arg,
+ &kf_dist_arg,
+ &scale_factors_arg,
+ &min_q_arg,
+ &max_q_arg,
+ &temporal_layers_arg,
+ &layering_mode_arg,
+ &threads_arg,
+ &aqmode_arg,
#if CONFIG_AV1_HIGHBITDEPTH
- &bitdepth_arg,
+ &bitdepth_arg,
#endif
- &speed_arg, &bitrates_arg, &dropframe_thresh_arg,
- &error_resilient_arg, &output_obu_arg, NULL
-};
+ &speed_arg,
+ &bitrates_arg,
+ &dropframe_thresh_arg,
+ &error_resilient_arg,
+ &output_obu_arg,
+ &test_decode_arg,
+ NULL };
#define zero(Dest) memset(&(Dest), 0, sizeof(Dest))
@@ -261,6 +278,7 @@
svc_params->number_temporal_layers = 1;
app_input->layering_mode = 0;
app_input->output_obu = 0;
+ app_input->decode = 1;
enc_cfg->g_threads = 1;
enc_cfg->rc_end_usage = AOM_CBR;
@@ -342,6 +360,11 @@
if (app_input->output_obu != 0 && app_input->output_obu != 1)
die("Invalid value for obu output flag (0, 1): %d.",
app_input->output_obu);
+ } else if (arg_match(&arg, &test_decode_arg, argi)) {
+ app_input->decode = arg_parse_uint(&arg);
+ if (app_input->decode != 0 && app_input->decode != 1)
+ die("Invalid value for test decode flag (0, 1): %d.",
+ app_input->decode);
} else {
++argj;
}
@@ -1250,8 +1273,10 @@
die("Failed to initialize encoder");
#if CONFIG_AV1_DECODER
- if (aom_codec_dec_init(&decoder, get_aom_decoder_by_index(0), NULL, 0)) {
- die("Failed to initialize decoder");
+ if (app_input.decode) {
+ if (aom_codec_dec_init(&decoder, get_aom_decoder_by_index(0), NULL, 0)) {
+ die("Failed to initialize decoder");
+ }
}
#endif
@@ -1460,9 +1485,11 @@
}
#if CONFIG_AV1_DECODER
- if (aom_codec_decode(&decoder, pkt->data.frame.buf,
- (unsigned int)pkt->data.frame.sz, NULL))
- die_codec(&decoder, "Failed to decode frame.");
+ if (app_input.decode) {
+ if (aom_codec_decode(&decoder, pkt->data.frame.buf,
+ (unsigned int)pkt->data.frame.sz, NULL))
+ die_codec(&decoder, "Failed to decode frame.");
+ }
#endif
break;
@@ -1470,12 +1497,14 @@
}
}
#if CONFIG_AV1_DECODER
- // Don't look for mismatch on top spatial and top temporal layers as they
- // are non reference frames.
- if ((ss_number_layers > 1 || ts_number_layers > 1) &&
- !(layer_id.temporal_layer_id > 0 &&
- layer_id.temporal_layer_id == (int)ts_number_layers - 1)) {
- test_decode(&codec, &decoder, frame_cnt, &mismatch_seen);
+ if (app_input.decode) {
+ // Don't look for mismatch on top spatial and top temporal layers as
+ // they are non reference frames.
+ if ((ss_number_layers > 1 || ts_number_layers > 1) &&
+ !(layer_id.temporal_layer_id > 0 &&
+ layer_id.temporal_layer_id == (int)ts_number_layers - 1)) {
+ test_decode(&codec, &decoder, frame_cnt, &mismatch_seen);
+ }
}
#endif
} // loop over spatial layers
diff --git a/libs.doxy_template b/libs.doxy_template
index 6e042ac..ba77751 100644
--- a/libs.doxy_template
+++ b/libs.doxy_template
@@ -103,14 +103,6 @@
OUTPUT_LANGUAGE = English
-# The OUTPUT_TEXT_DIRECTION tag is used to specify the direction in which all
-# documentation generated by doxygen is written. Doxygen will use this
-# information to generate all generated output in the proper direction.
-# Possible values are: None, LTR, RTL and Context.
-# The default value is: None.
-
-OUTPUT_TEXT_DIRECTION = None
-
# If the BRIEF_MEMBER_DESC tag is set to YES, doxygen will include brief member
# descriptions after the members that are listed in the file and class
# documentation (similar to Javadoc). Set to NO to disable this.
@@ -1820,16 +1812,6 @@
LATEX_HIDE_INDICES = NO
-# If the LATEX_SOURCE_CODE tag is set to YES then doxygen will include source
-# code with syntax highlighting in the LaTeX output.
-#
-# Note that which sources are shown also depends on other settings such as
-# SOURCE_BROWSER.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-LATEX_SOURCE_CODE = NO
-
# The LATEX_BIB_STYLE tag can be used to specify the style to use for the
# bibliography, e.g. plainnat, or ieeetr. See
# https://en.wikipedia.org/wiki/BibTeX and \cite for more info.
@@ -1910,16 +1892,6 @@
RTF_EXTENSIONS_FILE =
-# If the RTF_SOURCE_CODE tag is set to YES then doxygen will include source code
-# with syntax highlighting in the RTF output.
-#
-# Note that which sources are shown also depends on other settings such as
-# SOURCE_BROWSER.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_RTF is set to YES.
-
-RTF_SOURCE_CODE = NO
-
#---------------------------------------------------------------------------
# Configuration options related to the man page output
#---------------------------------------------------------------------------
@@ -1982,15 +1954,6 @@
DOCBOOK_OUTPUT = docbook
-# If the DOCBOOK_PROGRAMLISTING tag is set to YES, doxygen will include the
-# program listings (including syntax highlighting and cross-referencing
-# information) to the DOCBOOK output. Note that enabling this will significantly
-# increase the size of the DOCBOOK output.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_DOCBOOK is set to YES.
-
-DOCBOOK_PROGRAMLISTING = NO
-
#---------------------------------------------------------------------------
# Configuration options for the AutoGen Definitions output
#---------------------------------------------------------------------------
@@ -2172,15 +2135,6 @@
# Configuration options related to the dot tool
#---------------------------------------------------------------------------
-# If the CLASS_DIAGRAMS tag is set to YES, doxygen will generate a class diagram
-# (in HTML and LaTeX) for classes with base or super classes. Setting the tag to
-# NO turns the diagrams off. Note that this option also works with HAVE_DOT
-# disabled, but it is recommended to install and use dot, since it yields more
-# powerful graphs.
-# The default value is: YES.
-
-CLASS_DIAGRAMS = YES
-
# You can include diagrams made with dia in doxygen documentation. Doxygen will
# then run dia to produce the diagram and insert it in the documentation. The
# DIA_PATH tag allows you to specify the directory where the dia binary resides.
@@ -2237,11 +2191,14 @@
DOT_FONTPATH =
-# If the CLASS_GRAPH tag is set to YES then doxygen will generate a graph for
-# each documented class showing the direct and indirect inheritance relations.
-# Setting this tag to YES will force the CLASS_DIAGRAMS tag to NO.
+# If the CLASS_GRAPH tag is set to YES (or GRAPH) then doxygen will generate a
+# graph for each documented class showing the direct and indirect inheritance
+# relations. In case HAVE_DOT is set as well dot will be used to draw the graph,
+# otherwise the built-in generator will be used. If the CLASS_GRAPH tag is set
+# to TEXT the direct and indirect inheritance relations will be shown as texts /
+# links.
+# Possible values are: NO, YES, TEXT and GRAPH.
# The default value is: YES.
-# This tag requires that the tag HAVE_DOT is set to YES.
CLASS_GRAPH = YES
diff --git a/test/acm_random.h b/test/acm_random.h
index 8b1d51a..bc38ba4 100644
--- a/test/acm_random.h
+++ b/test/acm_random.h
@@ -27,43 +27,52 @@
void Reset(int seed) { random_.Reseed(seed); }
// Generates a random 31-bit unsigned integer from [0, 2^31).
- uint32_t Rand31(void) {
+ uint32_t Rand31() {
return random_.Generate(testing::internal::Random::kMaxRange);
}
- uint16_t Rand16(void) {
+ uint16_t Rand16() {
const uint32_t value =
random_.Generate(testing::internal::Random::kMaxRange);
+ // There's a bit more entropy in the upper bits of this implementation.
return (value >> 15) & 0xffff;
}
- int16_t Rand15Signed(void) {
+ int16_t Rand16Signed() { return static_cast<int16_t>(Rand16()); }
+
+ int16_t Rand15() {
const uint32_t value =
random_.Generate(testing::internal::Random::kMaxRange);
- return (value >> 17) & 0xffff;
+ // There's a bit more entropy in the upper bits of this implementation.
+ return (value >> 16) & 0x7fff;
}
- uint16_t Rand12(void) {
+ int16_t Rand15Signed() {
+ // Use 15 bits: values between 16383 (0x3FFF) and -16384 (0xC000).
+ return static_cast<int16_t>(Rand15()) - (1 << 14);
+ }
+
+ uint16_t Rand12() {
const uint32_t value =
random_.Generate(testing::internal::Random::kMaxRange);
// There's a bit more entropy in the upper bits of this implementation.
return (value >> 19) & 0xfff;
}
- int16_t Rand9Signed(void) {
+ int16_t Rand9Signed() {
// Use 9 bits: values between 255 (0x0FF) and -256 (0x100).
const uint32_t value = random_.Generate(512);
return static_cast<int16_t>(value) - 256;
}
- uint8_t Rand8(void) {
+ uint8_t Rand8() {
const uint32_t value =
random_.Generate(testing::internal::Random::kMaxRange);
// There's a bit more entropy in the upper bits of this implementation.
return (value >> 23) & 0xff;
}
- uint8_t Rand8Extremes(void) {
+ uint8_t Rand8Extremes() {
// Returns a random value near 0 or near 255, to better exercise
// saturation behavior.
const uint8_t r = Rand8();
@@ -74,7 +83,7 @@
int operator()(int n) { return PseudoUniform(n); }
- static int DeterministicSeed(void) { return 0xbaba; }
+ static int DeterministicSeed() { return 0xbaba; }
private:
testing::internal::Random random_;
diff --git a/test/aom_image_test.cc b/test/aom_image_test.cc
index 6ee0058..ad48e73 100644
--- a/test/aom_image_test.cc
+++ b/test/aom_image_test.cc
@@ -42,7 +42,9 @@
EXPECT_EQ(aom_img_set_rect(&img, 0, 0, kWidth, kHeight, 0), 0);
// This would result in overflow because -1 is cast to UINT_MAX.
- EXPECT_NE(aom_img_set_rect(&img, -1, -1, kWidth, kHeight, 0), 0);
+ EXPECT_NE(aom_img_set_rect(&img, static_cast<unsigned int>(-1),
+ static_cast<unsigned int>(-1), kWidth, kHeight, 0),
+ 0);
}
TEST(AomImageTest, AomImgAllocNv12) {
diff --git a/test/av1_quantize_test.cc b/test/av1_quantize_test.cc
index 731e99c..5823647 100644
--- a/test/av1_quantize_test.cc
+++ b/test/av1_quantize_test.cc
@@ -79,8 +79,8 @@
}
for (int j = 0; j < 2; j++) {
- zbin_ptr[j] = rnd.Rand16();
- quant_shift_ptr[j] = rnd.Rand16();
+ zbin_ptr[j] = rnd.Rand16Signed();
+ quant_shift_ptr[j] = rnd.Rand16Signed();
// int16_t positive
dequant_ptr[j] = abs(rnd(dequantRange));
quant_ptr[j] = static_cast<int16_t>((1 << 16) / dequant_ptr[j]);
@@ -155,8 +155,8 @@
coeff_ptr[rnd(count)] = rnd(coeffRange);
for (int j = 0; j < 2; j++) {
- zbin_ptr[j] = rnd.Rand16();
- quant_shift_ptr[j] = rnd.Rand16();
+ zbin_ptr[j] = rnd.Rand16Signed();
+ quant_shift_ptr[j] = rnd.Rand16Signed();
// int16_t positive
dequant_ptr[j] = abs(rnd(dequantRange));
quant_ptr[j] = (1 << 16) / dequant_ptr[j];
diff --git a/test/av1_wedge_utils_test.cc b/test/av1_wedge_utils_test.cc
index a51ce12..46f6d92 100644
--- a/test/av1_wedge_utils_test.cc
+++ b/test/av1_wedge_utils_test.cc
@@ -341,7 +341,7 @@
for (int iter = 0; iter < kIterations && !HasFatalFailure(); ++iter) {
for (int i = 0; i < MAX_SB_SQUARE; ++i) {
- a[i] = rng_.Rand16();
+ a[i] = rng_.Rand16Signed();
b[i] = rng_(2 * INT16_MAX + 1) - INT16_MAX;
}
@@ -374,6 +374,13 @@
av1_wedge_compute_delta_squares_sse2)));
#endif // HAVE_SSE2
+#if HAVE_NEON
+INSTANTIATE_TEST_SUITE_P(
+ NEON, WedgeUtilsSSEOptTest,
+ ::testing::Values(TestFuncsFSSE(av1_wedge_sse_from_residuals_c,
+ av1_wedge_sse_from_residuals_neon)));
+#endif // HAVE_NEON
+
#if HAVE_AVX2
INSTANTIATE_TEST_SUITE_P(
AVX2, WedgeUtilsSSEOptTest,
diff --git a/test/avg_test.cc b/test/avg_test.cc
index b12d1ef..93f4c34 100644
--- a/test/avg_test.cc
+++ b/test/avg_test.cc
@@ -636,7 +636,7 @@
}
TEST_P(VectorVarTest, DISABLED_Speed) {
FillRandom();
- const int numIter = 50000;
+ const int numIter = 5000000;
printf("Width = %d number of iteration is %d \n", width, numIter);
int sum_c_var = 0;
@@ -942,6 +942,16 @@
make_tuple(5, &aom_vector_var_c, &aom_vector_var_neon)));
#endif
+#if HAVE_SSE4_1
+INSTANTIATE_TEST_SUITE_P(
+ SSE4_1, VectorVarTest,
+ ::testing::Values(make_tuple(2, &aom_vector_var_c, &aom_vector_var_sse4_1),
+ make_tuple(3, &aom_vector_var_c, &aom_vector_var_sse4_1),
+ make_tuple(4, &aom_vector_var_c, &aom_vector_var_sse4_1),
+ make_tuple(5, &aom_vector_var_c,
+ &aom_vector_var_sse4_1)));
+#endif // HAVE_SSE4_1
+
#if HAVE_AVX2
INSTANTIATE_TEST_SUITE_P(
AVX2, SatdTest,
diff --git a/test/cfl_test.cc b/test/cfl_test.cc
index 98cc9ab..97533da 100644
--- a/test/cfl_test.cc
+++ b/test/cfl_test.cc
@@ -192,7 +192,7 @@
TEST_P(CFLSubAvgTest, SubAvgTest) {
for (int it = 0; it < NUM_ITERATIONS; it++) {
- randData(&ACMRandom::Rand15Signed);
+ randData(&ACMRandom::Rand15);
sub_avg((uint16_t *)data, data);
sub_avg_ref((uint16_t *)data_ref, data_ref);
assert_eq<int16_t>(data, data_ref, width, height);
@@ -202,7 +202,7 @@
TEST_P(CFLSubAvgTest, DISABLED_SubAvgSpeedTest) {
aom_usec_timer ref_timer;
aom_usec_timer timer;
- randData(&ACMRandom::Rand15Signed);
+ randData(&ACMRandom::Rand15);
aom_usec_timer_start(&ref_timer);
for (int k = 0; k < NUM_ITERATIONS_SPEED; k++) {
sub_avg_ref((uint16_t *)data_ref, data_ref);
diff --git a/test/datarate_test.cc b/test/datarate_test.cc
index ffa41e0..e59c876 100644
--- a/test/datarate_test.cc
+++ b/test/datarate_test.cc
@@ -399,7 +399,7 @@
ASSERT_NO_FATAL_FAILURE(RunLoop(&video));
ASSERT_GE(effective_datarate_, cfg_.rc_target_bitrate * 0.83)
<< " The datarate for the file is lower than target by too much!";
- ASSERT_LE(effective_datarate_, cfg_.rc_target_bitrate * 1.24)
+ ASSERT_LE(effective_datarate_, cfg_.rc_target_bitrate * 1.25)
<< " The datarate for the file is greater than target by too much!";
}
};
diff --git a/test/datarate_test.h b/test/datarate_test.h
index cb5d6e5..19b1b39 100644
--- a/test/datarate_test.h
+++ b/test/datarate_test.h
@@ -134,7 +134,7 @@
++tot_frame_number_;
}
- virtual void EndPassHook(void) {
+ virtual void EndPassHook() {
duration_ = (last_pts_ + 1) * timebase_;
// Effective file datarate:
effective_datarate_ = (bits_total_ / 1000.0) / duration_;
diff --git a/test/ducky_encode_test.cc b/test/ducky_encode_test.cc
index ce64253..fe4ddf3 100644
--- a/test/ducky_encode_test.cc
+++ b/test/ducky_encode_test.cc
@@ -35,7 +35,7 @@
1, "bus_352x288_420_f20_b8.yuv" };
video_info.file_path =
libaom_test::GetDataPath() + "/" + video_info.file_path;
- DuckyEncode ducky_encode(video_info, kMaxRefFrames);
+ DuckyEncode ducky_encode(video_info, kMaxRefFrames, 3, 128);
std::vector<FIRSTPASS_STATS> frame_stats =
ducky_encode.ComputeFirstPassStats();
EXPECT_EQ(frame_stats.size(), static_cast<size_t>(video_info.frame_count));
@@ -52,7 +52,7 @@
17, "bus_352x288_420_f20_b8.yuv" };
video_info.file_path =
libaom_test::GetDataPath() + "/" + video_info.file_path;
- DuckyEncode ducky_encode(video_info, kMaxRefFrames);
+ DuckyEncode ducky_encode(video_info, kMaxRefFrames, 3, 128);
std::vector<FIRSTPASS_STATS> frame_stats =
ducky_encode.ComputeFirstPassStats();
ducky_encode.StartEncode(frame_stats);
@@ -78,7 +78,7 @@
17, "bus_352x288_420_f20_b8.yuv" };
video_info.file_path =
libaom_test::GetDataPath() + "/" + video_info.file_path;
- DuckyEncode ducky_encode(video_info, kMaxRefFrames);
+ DuckyEncode ducky_encode(video_info, kMaxRefFrames, 3, 128);
std::vector<FIRSTPASS_STATS> frame_stats =
ducky_encode.ComputeFirstPassStats();
ducky_encode.StartEncode(frame_stats);
diff --git a/test/encode_api_test.cc b/test/encode_api_test.cc
index 2566abe..8303880 100644
--- a/test/encode_api_test.cc
+++ b/test/encode_api_test.cc
@@ -17,13 +17,14 @@
#include "aom/aomcx.h"
#include "aom/aom_encoder.h"
+#include "aom/aom_image.h"
namespace {
#if CONFIG_REALTIME_ONLY
-const int kUsage = 1;
+const int kUsage = AOM_USAGE_REALTIME;
#else
-const int kUsage = 0;
+const int kUsage = AOM_USAGE_GOOD_QUALITY;
#endif
TEST(EncodeAPI, InvalidParams) {
@@ -81,6 +82,30 @@
EXPECT_EQ(AOM_CODEC_OK, aom_codec_destroy(&enc));
}
+TEST(EncodeAPI, SetSFrameOnFirstFrame) {
+ constexpr int kWidth = 2;
+ constexpr int kHeight = 128;
+ unsigned char kBuffer[kWidth * kHeight * 3] = { 0 };
+ aom_image_t img;
+ ASSERT_EQ(aom_img_wrap(&img, AOM_IMG_FMT_I420, kWidth, kHeight, 1, kBuffer),
+ &img);
+
+ aom_codec_iface_t *iface = aom_codec_av1_cx();
+ aom_codec_enc_cfg_t cfg;
+ ASSERT_EQ(aom_codec_enc_config_default(iface, &cfg, kUsage), AOM_CODEC_OK);
+ cfg.g_w = kWidth;
+ cfg.g_h = kHeight;
+
+ aom_codec_ctx_t enc;
+ ASSERT_EQ(aom_codec_enc_init(&enc, iface, &cfg, 0), AOM_CODEC_OK);
+ // One of these aom_codec_encode() calls should fail.
+ if (aom_codec_encode(&enc, &img, 0, 1, AOM_EFLAG_SET_S_FRAME) ==
+ AOM_CODEC_OK) {
+ EXPECT_NE(aom_codec_encode(&enc, NULL, 0, 0, 0), AOM_CODEC_OK);
+ }
+ EXPECT_EQ(aom_codec_destroy(&enc), AOM_CODEC_OK);
+}
+
#if !CONFIG_REALTIME_ONLY
TEST(EncodeAPI, AllIntraMode) {
aom_codec_iface_t *iface = aom_codec_av1_cx();
diff --git a/test/encodetxb_test.cc b/test/encodetxb_test.cc
index ee09ea6..c1b6709 100644
--- a/test/encodetxb_test.cc
+++ b/test/encodetxb_test.cc
@@ -229,7 +229,7 @@
ACMRandom rnd(ACMRandom::DeterministicSeed());
for (int i = 0; i < width * height; i++) {
- coeff[i] = rnd.Rand15Signed() + rnd.Rand15Signed();
+ coeff[i] = rnd.Rand16Signed();
}
for (int i = 0; i < TX_PAD_2D; i++) {
levels_buf[0][i] = rnd.Rand8();
diff --git a/test/force_key_frame_test.cc b/test/force_key_frame_test.cc
new file mode 100644
index 0000000..78662ed
--- /dev/null
+++ b/test/force_key_frame_test.cc
@@ -0,0 +1,89 @@
+/*
+ * Copyright (c) 2022, Alliance for Open Media. All rights reserved
+ *
+ * This source code is subject to the terms of the BSD 2 Clause License and
+ * the Alliance for Open Media Patent License 1.0. If the BSD 2 Clause License
+ * was not distributed with this source code in the LICENSE file, you can
+ * obtain it at www.aomedia.org/license/software. If the Alliance for Open
+ * Media Patent License 1.0 was not distributed with this source code in the
+ * PATENTS file, you can obtain it at www.aomedia.org/license/patent.
+ */
+
+// Tests for https://crbug.com/aomedia/3327.
+//
+// In good-quality mode, set cfg.g_lag_in_frames to 1 or 0 and encode two
+// frames in one-pass mode. Pass AOM_EFLAG_FORCE_KF to the second
+// aom_codec_encode() call. Both frames should be encoded as key frames.
+
+#include <memory>
+
+#include "aom/aomcx.h"
+#include "aom/aom_encoder.h"
+#include "third_party/googletest/src/googletest/include/gtest/gtest.h"
+
+namespace {
+
+TEST(ForceKeyFrameTest, OnePassMode) {
+ // A buffer of gray samples of size 128x128, YUV 4:2:0.
+ constexpr size_t kImageDataSize = 128 * 128 + 2 * 64 * 64;
+ std::unique_ptr<unsigned char[]> img_data(new unsigned char[kImageDataSize]);
+ ASSERT_NE(img_data, nullptr);
+ memset(img_data.get(), 128, kImageDataSize);
+
+ aom_codec_iface_t *iface = aom_codec_av1_cx();
+ aom_codec_enc_cfg_t cfg;
+ ASSERT_EQ(AOM_CODEC_OK,
+ aom_codec_enc_config_default(iface, &cfg, AOM_USAGE_GOOD_QUALITY));
+ cfg.g_w = 128;
+ cfg.g_h = 128;
+ cfg.g_pass = AOM_RC_ONE_PASS;
+ // TODO(crbug.com/aomedia/3327): Lower cfg.g_lag_in_frames to 1 or 0 to see
+ // the bug.
+ cfg.g_lag_in_frames = 2;
+ aom_codec_ctx_t enc;
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_enc_init(&enc, iface, &cfg, 0));
+
+ aom_image_t img;
+ EXPECT_EQ(&img,
+ aom_img_wrap(&img, AOM_IMG_FMT_I420, 128, 128, 1, img_data.get()));
+
+ aom_codec_iter_t iter;
+ const aom_codec_cx_pkt_t *pkt;
+ int frame_count = 0;
+
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_encode(&enc, &img, 0, 1, 0));
+
+ iter = nullptr;
+ while ((pkt = aom_codec_get_cx_data(&enc, &iter)) != nullptr) {
+ ASSERT_EQ(pkt->kind, AOM_CODEC_CX_FRAME_PKT);
+ EXPECT_NE(pkt->data.frame.flags & AOM_FRAME_IS_KEY, 0u)
+ << "frame " << frame_count;
+ frame_count++;
+ }
+
+ EXPECT_EQ(AOM_CODEC_OK,
+ aom_codec_encode(&enc, &img, 1, 1, AOM_EFLAG_FORCE_KF));
+
+ iter = nullptr;
+ while ((pkt = aom_codec_get_cx_data(&enc, &iter)) != nullptr) {
+ ASSERT_EQ(pkt->kind, AOM_CODEC_CX_FRAME_PKT);
+ EXPECT_NE(pkt->data.frame.flags & AOM_FRAME_IS_KEY, 0u)
+ << "frame " << frame_count;
+ frame_count++;
+ }
+
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_encode(&enc, nullptr, 0, 0, 0));
+
+ iter = nullptr;
+ while ((pkt = aom_codec_get_cx_data(&enc, &iter)) != nullptr) {
+ ASSERT_EQ(pkt->kind, AOM_CODEC_CX_FRAME_PKT);
+ EXPECT_NE(pkt->data.frame.flags & AOM_FRAME_IS_KEY, 0u)
+ << "frame " << frame_count;
+ frame_count++;
+ }
+
+ EXPECT_EQ(frame_count, 2);
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_destroy(&enc));
+}
+
+} // namespace
diff --git a/test/forced_max_frame_width_height_test.cc b/test/forced_max_frame_width_height_test.cc
new file mode 100644
index 0000000..1f45328
--- /dev/null
+++ b/test/forced_max_frame_width_height_test.cc
@@ -0,0 +1,172 @@
+/*
+ * Copyright (c) 2022, Alliance for Open Media. All rights reserved
+ *
+ * This source code is subject to the terms of the BSD 2 Clause License and
+ * the Alliance for Open Media Patent License 1.0. If the BSD 2 Clause License
+ * was not distributed with this source code in the LICENSE file, you can
+ * obtain it at www.aomedia.org/license/software. If the Alliance for Open
+ * Media Patent License 1.0 was not distributed with this source code in the
+ * PATENTS file, you can obtain it at www.aomedia.org/license/patent.
+ */
+
+// Tests for https://crbug.com/aomedia/3326.
+//
+// Set cfg.g_forced_max_frame_width and cfg.g_forced_max_frame_height and
+// encode two frames of increasing sizes. The second aom_codec_encode() should
+// not crash or have memory errors.
+
+#include <memory>
+
+#include "aom/aomcx.h"
+#include "aom/aom_encoder.h"
+#include "third_party/googletest/src/googletest/include/gtest/gtest.h"
+
+namespace {
+
+// cfg.g_lag_in_frames must be set to 0 or 1 to allow the frame size to change,
+// as required by the following check in encoder_set_config() in
+// av1/av1_cx_iface.c:
+//
+// if (cfg->g_w != ctx->cfg.g_w || cfg->g_h != ctx->cfg.g_h) {
+// if (cfg->g_lag_in_frames > 1 || cfg->g_pass != AOM_RC_ONE_PASS)
+// ERROR("Cannot change width or height after initialization");
+// ...
+// }
+
+void RunTest(unsigned int usage, unsigned int lag_in_frames,
+ const char *tune_metric) {
+ // A buffer of gray samples. Large enough for 128x128 and 256x256, YUV 4:2:0.
+ constexpr size_t kImageDataSize = 256 * 256 + 2 * 128 * 128;
+ std::unique_ptr<unsigned char[]> img_data(new unsigned char[kImageDataSize]);
+ ASSERT_NE(img_data, nullptr);
+ memset(img_data.get(), 128, kImageDataSize);
+
+ aom_codec_iface_t *iface = aom_codec_av1_cx();
+ aom_codec_enc_cfg_t cfg;
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_enc_config_default(iface, &cfg, usage));
+ cfg.g_w = 128;
+ cfg.g_h = 128;
+ cfg.g_forced_max_frame_width = 256;
+ cfg.g_forced_max_frame_height = 256;
+ cfg.g_lag_in_frames = lag_in_frames;
+ aom_codec_ctx_t enc;
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_enc_init(&enc, iface, &cfg, 0));
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_set_option(&enc, "tune", tune_metric));
+
+ aom_image_t img;
+ EXPECT_EQ(&img,
+ aom_img_wrap(&img, AOM_IMG_FMT_I420, 128, 128, 1, img_data.get()));
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_encode(&enc, &img, 0, 1, 0));
+
+ cfg.g_w = 256;
+ cfg.g_h = 256;
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_enc_config_set(&enc, &cfg));
+
+ EXPECT_EQ(&img,
+ aom_img_wrap(&img, AOM_IMG_FMT_I420, 256, 256, 1, img_data.get()));
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_encode(&enc, &img, 0, 1, 0));
+
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_encode(&enc, nullptr, 0, 0, 0));
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_destroy(&enc));
+}
+
+#if !CONFIG_REALTIME_ONLY
+
+TEST(EncodeForcedMaxFrameWidthHeight, DISABLED_GoodQualityLag0TunePSNR) {
+ RunTest(AOM_USAGE_GOOD_QUALITY, /*lag_in_frames=*/0, "psnr");
+}
+
+TEST(EncodeForcedMaxFrameWidthHeight, DISABLED_GoodQualityLag0TuneSSIM) {
+ RunTest(AOM_USAGE_GOOD_QUALITY, /*lag_in_frames=*/0, "ssim");
+}
+
+TEST(EncodeForcedMaxFrameWidthHeight, DISABLED_GoodQualityLag1TunePSNR) {
+ RunTest(AOM_USAGE_GOOD_QUALITY, /*lag_in_frames=*/1, "psnr");
+}
+
+TEST(EncodeForcedMaxFrameWidthHeight, DISABLED_GoodQualityLag1TuneSSIM) {
+ RunTest(AOM_USAGE_GOOD_QUALITY, /*lag_in_frames=*/1, "ssim");
+}
+
+#endif // !CONFIG_REALTIME_ONLY
+
+TEST(EncodeForcedMaxFrameWidthHeight, RealtimeLag0TunePSNR) {
+ RunTest(AOM_USAGE_REALTIME, /*lag_in_frames=*/0, "psnr");
+}
+
+TEST(EncodeForcedMaxFrameWidthHeight, RealtimeLag0TuneSSIM) {
+ RunTest(AOM_USAGE_REALTIME, /*lag_in_frames=*/0, "ssim");
+}
+
+TEST(EncodeForcedMaxFrameWidthHeight, RealtimeLag1TunePSNR) {
+ RunTest(AOM_USAGE_REALTIME, /*lag_in_frames=*/1, "psnr");
+}
+
+TEST(EncodeForcedMaxFrameWidthHeight, RealtimeLag1TuneSSIM) {
+ RunTest(AOM_USAGE_REALTIME, /*lag_in_frames=*/1, "ssim");
+}
+
+TEST(EncodeForcedMaxFrameWidthHeight, MaxFrameSizeTooBig) {
+ aom_codec_iface_t *iface = aom_codec_av1_cx();
+ aom_codec_enc_cfg_t cfg;
+ EXPECT_EQ(AOM_CODEC_OK,
+ aom_codec_enc_config_default(iface, &cfg, AOM_USAGE_REALTIME));
+ cfg.g_w = 256;
+ cfg.g_h = 256;
+ cfg.g_forced_max_frame_width = 131072;
+ cfg.g_forced_max_frame_height = 131072;
+ aom_codec_ctx_t enc;
+ EXPECT_EQ(AOM_CODEC_INVALID_PARAM, aom_codec_enc_init(&enc, iface, &cfg, 0));
+}
+
+TEST(EncodeForcedMaxFrameWidthHeight, FirstFrameTooBig) {
+ aom_codec_iface_t *iface = aom_codec_av1_cx();
+ aom_codec_enc_cfg_t cfg;
+ EXPECT_EQ(AOM_CODEC_OK,
+ aom_codec_enc_config_default(iface, &cfg, AOM_USAGE_REALTIME));
+ cfg.g_w = 258;
+ cfg.g_h = 256;
+ cfg.g_forced_max_frame_width = 256;
+ cfg.g_forced_max_frame_height = 256;
+ aom_codec_ctx_t enc;
+ EXPECT_EQ(AOM_CODEC_INVALID_PARAM, aom_codec_enc_init(&enc, iface, &cfg, 0));
+ cfg.g_w = 256;
+ cfg.g_h = 258;
+ EXPECT_EQ(AOM_CODEC_INVALID_PARAM, aom_codec_enc_init(&enc, iface, &cfg, 0));
+ cfg.g_w = 256;
+ cfg.g_h = 256;
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_enc_init(&enc, iface, &cfg, 0));
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_destroy(&enc));
+}
+
+TEST(EncodeForcedMaxFrameWidthHeight, SecondFrameTooBig) {
+ // A buffer of gray samples. Large enough for 128x128 and 256x256, YUV 4:2:0.
+ constexpr size_t kImageDataSize = 256 * 256 + 2 * 128 * 128;
+ std::unique_ptr<unsigned char[]> img_data(new unsigned char[kImageDataSize]);
+ ASSERT_NE(img_data, nullptr);
+ memset(img_data.get(), 128, kImageDataSize);
+
+ aom_codec_iface_t *iface = aom_codec_av1_cx();
+ aom_codec_enc_cfg_t cfg;
+ EXPECT_EQ(AOM_CODEC_OK,
+ aom_codec_enc_config_default(iface, &cfg, AOM_USAGE_REALTIME));
+ cfg.g_w = 128;
+ cfg.g_h = 128;
+ cfg.g_forced_max_frame_width = 255;
+ cfg.g_forced_max_frame_height = 256;
+ aom_codec_ctx_t enc;
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_enc_init(&enc, iface, &cfg, 0));
+
+ aom_image_t img;
+ EXPECT_EQ(&img,
+ aom_img_wrap(&img, AOM_IMG_FMT_I420, 128, 128, 1, img_data.get()));
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_encode(&enc, &img, 0, 1, 0));
+
+ cfg.g_w = 256;
+ cfg.g_h = 256;
+ EXPECT_EQ(AOM_CODEC_INVALID_PARAM, aom_codec_enc_config_set(&enc, &cfg));
+
+ EXPECT_EQ(AOM_CODEC_OK, aom_codec_destroy(&enc));
+}
+
+} // namespace
diff --git a/test/hash_test.cc b/test/hash_test.cc
index 5ce0fbb..61e0b51 100644
--- a/test/hash_test.cc
+++ b/test/hash_test.cc
@@ -131,4 +131,11 @@
::testing::ValuesIn(kValidBlockSize)));
#endif
+#if HAVE_ARM_CRC32
+INSTANTIATE_TEST_SUITE_P(
+ ARM_CRC32, AV1Crc32cHashTest,
+ ::testing::Combine(::testing::Values(&av1_get_crc32c_value_arm_crc32),
+ ::testing::ValuesIn(kValidBlockSize)));
+#endif
+
} // namespace
diff --git a/test/horver_correlation_test.cc b/test/horver_correlation_test.cc
index d1fd578..2873490 100644
--- a/test/horver_correlation_test.cc
+++ b/test/horver_correlation_test.cc
@@ -39,8 +39,8 @@
target_func_ = GET_PARAM(0);
}
virtual void TearDown() { aom_free(data_buf_); }
- void RunHorverTest(void);
- void RunHorverTest_ExtremeValues(void);
+ void RunHorverTest();
+ void RunHorverTest_ExtremeValues();
void RunHorverSpeedTest(int run_times);
private:
@@ -50,7 +50,7 @@
};
GTEST_ALLOW_UNINSTANTIATED_PARAMETERIZED_TEST(HorverTest);
-void HorverTest::RunHorverTest(void) {
+void HorverTest::RunHorverTest() {
for (int block_size = 0; block_size < BLOCK_SIZES_ALL; block_size++) {
const int w = block_size_wide[block_size];
const int h = block_size_high[block_size];
@@ -107,7 +107,7 @@
}
}
-void HorverTest::RunHorverTest_ExtremeValues(void) {
+void HorverTest::RunHorverTest_ExtremeValues() {
for (int i = 0; i < MAX_SB_SQUARE; ++i) {
// Most of get_horver_test is squaring and summing, so simply saturating
// the whole buffer is mostly likely to cause an overflow.
diff --git a/test/horz_superres_test.cc b/test/horz_superres_test.cc
index 12e14dc..323aa93 100644
--- a/test/horz_superres_test.cc
+++ b/test/horz_superres_test.cc
@@ -56,7 +56,7 @@
45.0 },
#if CONFIG_AV1_HIGHBITDEPTH
{ "park_joy_90p_10_444.y4m", AOM_IMG_FMT_I44416, AOM_BITS_10, 1, 5, 0, 27.0,
- 48.0 },
+ 47.9 },
#endif
{ "screendata.y4m", AOM_IMG_FMT_I420, AOM_BITS_8, 0, 4, 1, 23.0, 56.0 },
// Image coding (single frame).
diff --git a/test/md5_helper.h b/test/md5_helper.h
index 9443cb2..69f1ae7 100644
--- a/test/md5_helper.h
+++ b/test/md5_helper.h
@@ -48,7 +48,7 @@
MD5Update(&md5_, data, static_cast<uint32_t>(size));
}
- const char *Get(void) {
+ const char *Get() {
static const char hex[16] = {
'0', '1', '2', '3', '4', '5', '6', '7',
'8', '9', 'a', 'b', 'c', 'd', 'e', 'f',
diff --git a/test/mock_ratectrl_qmode.h b/test/mock_ratectrl_qmode.h
index 2dbd8ef..3b6f290 100644
--- a/test/mock_ratectrl_qmode.h
+++ b/test/mock_ratectrl_qmode.h
@@ -25,6 +25,7 @@
(const FirstpassInfo &firstpass_info), (override));
MOCK_METHOD(StatusOr<GopEncodeInfo>, GetGopEncodeInfo,
(const GopStruct &gop_struct, const TplGopStats &tpl_gop_stats,
+ const std::vector<LookaheadStats> &lookahead_stats,
const RefFrameTable &ref_frame_table_snapshot_init),
(override));
};
diff --git a/test/ratectrl_qmode_test.cc b/test/ratectrl_qmode_test.cc
index d950626..9218ef9 100644
--- a/test/ratectrl_qmode_test.cc
+++ b/test/ratectrl_qmode_test.cc
@@ -35,6 +35,7 @@
constexpr int kRefFrameTableSize = 7;
constexpr int kFrameWidth = 352;
constexpr int kFrameHeight = 288;
+constexpr int kFrameLimit = 250;
MATCHER(IsOkStatus, "") {
*result_listener << "with code " << arg.code
@@ -60,7 +61,8 @@
}
void ReadFirstpassInfo(const std::string &filename,
- aom::FirstpassInfo *firstpass_info) {
+ aom::FirstpassInfo *firstpass_info,
+ const int frame_limit) {
// These golden files are generated by the following command line:
// ./aomenc --width=352 --height=288 --fps=30/1 --limit=250 --codec=av1
// --cpu-used=3 --end-usage=q --cq-level=36 --threads=0 --profile=0
@@ -80,7 +82,9 @@
firstpass_info->num_mbs_16x16 =
(kFrameWidth / 16 + 1) * (kFrameHeight / 16 + 1);
std::string newline;
- while (std::getline(firstpass_stats_file, newline)) {
+ int frame_number = 0;
+ while (std::getline(firstpass_stats_file, newline) &&
+ frame_number < frame_limit) {
std::istringstream iss(newline);
FIRSTPASS_STATS firstpass_stats_input = {};
ASSERT_EQ(ReadDouble(iss, &firstpass_stats_input.frame), "");
@@ -115,6 +119,8 @@
<< firstpass_info->stats_list.size() + 1 << "\n"
<< newline;
firstpass_info->stats_list.push_back(firstpass_stats_input);
+
+ frame_number++;
}
}
} // namespace
@@ -226,7 +232,6 @@
rc_param_.min_gop_show_frame_count = 4;
rc_param_.ref_frame_table_size = 7;
rc_param_.max_ref_frames = 7;
- rc_param_.max_depth = 5;
rc_param_.base_q_index = 128;
rc_param_.frame_height = kFrameHeight;
rc_param_.frame_width = kFrameWidth;
@@ -240,7 +245,7 @@
const bool has_key_frame = false;
const int global_coding_idx_offset = 5;
const int global_order_idx_offset = 20;
- RefFrameManager ref_frame_manager(kRefFrameTableSize);
+ RefFrameManager ref_frame_manager(kRefFrameTableSize, 7);
GopStruct gop_struct =
ConstructGop(&ref_frame_manager, show_frame_count, has_key_frame,
global_coding_idx_offset, global_order_idx_offset);
@@ -249,8 +254,7 @@
TestGopGlobalOrderIdx(gop_struct, global_order_idx_offset);
TestGopGlobalCodingIdx(gop_struct, global_coding_idx_offset);
TestColocatedShowFrame(gop_struct);
- const int max_layer_depth =
- ref_frame_manager.ForwardMaxSize() + kLayerDepthOffset;
+ const int max_layer_depth = ref_frame_manager.MaxRefFrame();
TestLayerDepth(gop_struct, max_layer_depth);
TestArfInterval(gop_struct);
}
@@ -260,7 +264,7 @@
const bool has_key_frame = true;
const int global_coding_idx_offset = 10;
const int global_order_idx_offset = 8;
- RefFrameManager ref_frame_manager(kRefFrameTableSize);
+ RefFrameManager ref_frame_manager(kRefFrameTableSize, 7);
GopStruct gop_struct =
ConstructGop(&ref_frame_manager, show_frame_count, has_key_frame,
global_coding_idx_offset, global_order_idx_offset);
@@ -269,8 +273,7 @@
TestGopGlobalOrderIdx(gop_struct, global_order_idx_offset);
TestGopGlobalCodingIdx(gop_struct, global_coding_idx_offset);
TestColocatedShowFrame(gop_struct);
- const int max_layer_depth =
- ref_frame_manager.ForwardMaxSize() + kLayerDepthOffset;
+ const int max_layer_depth = ref_frame_manager.MaxRefFrame();
TestLayerDepth(gop_struct, max_layer_depth);
TestArfInterval(gop_struct);
}
@@ -280,7 +283,7 @@
const bool has_key_frame = false;
const int global_coding_idx_offset = 5;
const int global_order_idx_offset = 20;
- RefFrameManager ref_frame_manager(kRefFrameTableSize);
+ RefFrameManager ref_frame_manager(kRefFrameTableSize, 7);
GopStruct gop_struct =
ConstructGop(&ref_frame_manager, show_frame_count, has_key_frame,
global_coding_idx_offset, global_order_idx_offset);
@@ -316,6 +319,7 @@
frame_stats.min_block_size = min_block_size;
frame_stats.frame_height = max_h * count;
frame_stats.frame_width = max_w * count;
+ frame_stats.rate_dist_present = false;
for (int i = 0; i < count; ++i) {
for (int j = 0; j < count; ++j) {
int h = max_h >> i;
@@ -578,6 +582,7 @@
}
}
+// TODO(jianj): Add tests for non empty lookahead stats.
TEST_F(RateControlQModeTest, ComputeTplGopDepStats) {
TplGopStats tpl_gop_stats;
std::vector<RefFrameTable> ref_frame_table_list;
@@ -595,7 +600,7 @@
ref_frame_table_list.push_back(CreateToyRefFrameTable(i));
}
const StatusOr<TplGopDepStats> gop_dep_stats =
- ComputeTplGopDepStats(tpl_gop_stats, ref_frame_table_list);
+ ComputeTplGopDepStats(tpl_gop_stats, {}, ref_frame_table_list);
ASSERT_THAT(gop_dep_stats.status(), IsOkStatus());
double expected_sum = 0;
@@ -622,7 +627,7 @@
GopFrameType::kRegularLeaf,
GopFrameType::kOverlay
};
- RefFrameManager ref_manager(kRefFrameTableSize);
+ RefFrameManager ref_manager(kRefFrameTableSize, 7);
int coding_idx = 0;
const int first_leaf_idx = 3;
EXPECT_EQ(type_list[first_leaf_idx], GopFrameType::kRegularLeaf);
@@ -654,8 +659,8 @@
// After the first kShowExisting, the kIntermediateArf should be moved from
// kForward to kLast due to the cur_global_order_idx_ update
EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kForward), 1);
- EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kBackward), 1);
- EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kLast), 2);
+ EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kBackward), 2);
+ EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kLast), 1);
const int second_leaf_idx = 5;
EXPECT_EQ(type_list[second_leaf_idx], GopFrameType::kRegularLeaf);
@@ -669,8 +674,8 @@
EXPECT_EQ(ref_manager.CurGlobalOrderIdx(), 3);
// An additional kRegularLeaf frame is added into kLast
EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kForward), 1);
- EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kBackward), 1);
- EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kLast), 3);
+ EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kBackward), 2);
+ EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kLast), 2);
const int first_overlay_idx = 6;
EXPECT_EQ(type_list[first_overlay_idx], GopFrameType::kOverlay);
@@ -686,8 +691,8 @@
// After the kOverlay, the kRegularArf should be moved from
// kForward to kBackward due to the cur_global_order_idx_ update
EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kForward), 0);
- EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kBackward), 2);
- EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kLast), 3);
+ EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kBackward), 3);
+ EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kLast), 2);
}
void TestRefFrameManagerPriority(const RefFrameManager &ref_manager,
@@ -723,7 +728,7 @@
GopFrameType::kRegularLeaf,
GopFrameType::kOverlay
};
- RefFrameManager ref_manager(kRefFrameTableSize);
+ RefFrameManager ref_manager(kRefFrameTableSize, 7);
int coding_idx = 0;
const int first_leaf_idx = 3;
EXPECT_EQ(type_list[first_leaf_idx], GopFrameType::kRegularLeaf);
@@ -746,9 +751,9 @@
ref_manager.UpdateRefFrameTable(&gop_frame);
}
- EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kBackward), 2);
+ EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kBackward), 3);
TestRefFrameManagerPriority(ref_manager, RefUpdateType::kBackward);
- EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kLast), 3);
+ EXPECT_EQ(ref_manager.GetRefFrameCountByType(RefUpdateType::kLast), 2);
TestRefFrameManagerPriority(ref_manager, RefUpdateType::kLast);
}
@@ -759,7 +764,7 @@
GopFrameType::kRegularArf,
GopFrameType::kIntermediateArf,
GopFrameType::kRegularLeaf };
- RefFrameManager ref_manager(kRefFrameTableSize);
+ RefFrameManager ref_manager(kRefFrameTableSize, 7);
for (int coding_idx = 0; coding_idx < frame_count; ++coding_idx) {
GopFrame gop_frame =
GopFrameBasic(0, 0, coding_idx, order_idx_list[coding_idx], 0, 0,
@@ -794,7 +799,7 @@
GopFrameType::kIntermediateArf,
GopFrameType::kRegularLeaf };
const std::vector<int> layer_depth_list = { 0, 2, 4, 6 };
- RefFrameManager ref_manager(kRefFrameTableSize);
+ RefFrameManager ref_manager(kRefFrameTableSize, 7);
for (int coding_idx = 0; coding_idx < frame_count; ++coding_idx) {
GopFrame gop_frame =
GopFrameBasic(0, 0, coding_idx, order_idx_list[coding_idx],
@@ -808,6 +813,7 @@
// Set different frame type
GopFrameType type = type_list[(i + 1) % frame_count];
GopFrame gop_frame = GopFrameBasic(0, 0, 0, 0, layer_depth, 0, type);
+ gop_frame.ref_frame_list = ref_manager.GetRefFrameListByPriority();
ReferenceFrame ref_frame = ref_manager.GetPrimaryRefFrame(gop_frame);
GopFrame primary_ref_frame =
ref_manager.GetRefFrameByIndex(ref_frame.index);
@@ -823,6 +829,7 @@
// Let the frame layer_depth sit in the middle of two reference frames
int layer_depth = mid_layer_depth_list[i];
GopFrame gop_frame = GopFrameBasic(0, 0, 0, 0, layer_depth, 0, type);
+ gop_frame.ref_frame_list = ref_manager.GetRefFrameListByPriority();
ReferenceFrame ref_frame = ref_manager.GetPrimaryRefFrame(gop_frame);
GopFrame primary_ref_frame =
ref_manager.GetRefFrameByIndex(ref_frame.index);
@@ -836,7 +843,7 @@
FirstpassInfo firstpass_info;
const std::string kFirstpassStatsFile = "firstpass_stats";
ASSERT_NO_FATAL_FAILURE(
- ReadFirstpassInfo(kFirstpassStatsFile, &firstpass_info));
+ ReadFirstpassInfo(kFirstpassStatsFile, &firstpass_info, kFrameLimit));
EXPECT_THAT(GetKeyFrameList(firstpass_info),
ElementsAre(0, 30, 60, 90, 120, 150, 180, 210, 240));
}
@@ -918,13 +925,6 @@
HasSubstr("max_ref_frames (8) must be in the range"));
}
-TEST_F(RateControlQModeTest, TestInvalidMaxDepth) {
- rc_param_.max_depth = 6;
- Status status = AV1RateControlQMode().SetRcParam(rc_param_);
- EXPECT_EQ(status.code, AOM_CODEC_INVALID_PARAM);
- EXPECT_THAT(status.message, HasSubstr("max_depth (6) must be in the range"));
-}
-
TEST_F(RateControlQModeTest, TestInvalidBaseQIndex) {
rc_param_.base_q_index = 256;
Status status = AV1RateControlQMode().SetRcParam(rc_param_);
@@ -963,7 +963,7 @@
// For the first GOP only, GetRefFrameTableList can be passed a
// default-constructed RefFrameTable (because it's all going to be
// replaced by the key frame anyway).
- rc.GetRefFrameTableList(gop_struct, RefFrameTable()),
+ rc.GetRefFrameTableList(gop_struct, {}, RefFrameTable()),
ElementsAre(
ElementsAre(matches_invalid, matches_invalid, matches_invalid),
ElementsAre(matches_frame0, matches_frame0, matches_frame0),
@@ -993,7 +993,7 @@
gop_struct.global_coding_idx_offset = 5; // This is not the first GOP.
gop_struct.gop_frame_list = { frame0, frame1, frame2 };
ASSERT_THAT(
- rc.GetRefFrameTableList(gop_struct, RefFrameTable(3, previous)),
+ rc.GetRefFrameTableList(gop_struct, {}, RefFrameTable(3, previous)),
ElementsAre(
ElementsAre(matches_previous, matches_previous, matches_previous),
ElementsAre(matches_previous, matches_previous, matches_frame0),
@@ -1004,7 +1004,7 @@
TEST_F(RateControlQModeTest, TestGopIntervals) {
FirstpassInfo firstpass_info;
ASSERT_NO_FATAL_FAILURE(
- ReadFirstpassInfo("firstpass_stats", &firstpass_info));
+ ReadFirstpassInfo("firstpass_stats", &firstpass_info, kFrameLimit));
AV1RateControlQMode rc;
ASSERT_THAT(rc.SetRcParam(rc_param_), IsOkStatus());
@@ -1018,12 +1018,15 @@
ElementsAre(21, 9, 30, 30, 16, 14, 21, 9, 30, 12, 16, 2, 30, 10));
}
+// TODO(b/242892473): Add a test which passes lookahead GOPs.
TEST_F(RateControlQModeTest, TestGetGopEncodeInfo) {
FirstpassInfo firstpass_info;
ASSERT_NO_FATAL_FAILURE(
- ReadFirstpassInfo("firstpass_stats", &firstpass_info));
+ ReadFirstpassInfo("firstpass_stats", &firstpass_info, 50));
AV1RateControlQMode rc;
rc_param_.max_gop_show_frame_count = 16;
+ rc_param_.max_ref_frames = 3;
+ rc_param_.base_q_index = 117;
ASSERT_THAT(rc.SetRcParam(rc_param_), IsOkStatus());
const auto gop_info = rc.DetermineGopInfo(firstpass_info);
ASSERT_THAT(gop_info.status(), IsOkStatus());
@@ -1032,9 +1035,10 @@
const aom::VideoInfo input_video = {
kFrameWidth, kFrameHeight,
frame_rate, AOM_IMG_FMT_I420,
- 250, libaom_test::GetDataPath() + "/hantro_collage_w352h288.yuv"
+ 50, libaom_test::GetDataPath() + "/hantro_collage_w352h288.yuv"
};
- DuckyEncode ducky_encode(input_video, 3, 3);
+ DuckyEncode ducky_encode(input_video, rc_param_.max_ref_frames, 3,
+ rc_param_.base_q_index);
ducky_encode.StartEncode(firstpass_info.stats_list);
// Read TPL stats
std::vector<TplGopStats> tpl_gop_list =
@@ -1045,12 +1049,16 @@
for (size_t gop_idx = 0; gop_idx < gop_list.size(); gop_idx++) {
size_t tpl_gop_idx = gop_idx - num_gop_skipped;
const auto gop_encode_info = rc.GetGopEncodeInfo(
- gop_list[gop_idx], tpl_gop_list[tpl_gop_idx], ref_frame_table);
+ gop_list[gop_idx], tpl_gop_list[tpl_gop_idx], {}, ref_frame_table);
ASSERT_THAT(gop_encode_info.status(), IsOkStatus());
for (auto &frame_param : gop_encode_info->param_list) {
- std::cout << frame_param.q_index << std::endl;
+ EXPECT_LE(frame_param.q_index, rc_param_.base_q_index);
}
ref_frame_table = gop_encode_info->final_snapshot;
+ for (auto &gop_frame : ref_frame_table) {
+ EXPECT_LE(static_cast<int>(gop_frame.ref_frame_list.size()),
+ rc_param_.max_ref_frames);
+ }
}
}
@@ -1062,7 +1070,8 @@
5, CreateToyTplFrameStatsWithDiffSizes(8, 8));
AV1RateControlQMode rc;
const Status status =
- rc.GetGopEncodeInfo(gop_struct, tpl_gop_stats, RefFrameTable()).status();
+ rc.GetGopEncodeInfo(gop_struct, tpl_gop_stats, {}, RefFrameTable())
+ .status();
EXPECT_EQ(status.code, AOM_CODEC_INVALID_PARAM);
EXPECT_THAT(status.message,
HasSubstr("Frame count of GopStruct (7) doesn't match frame "
@@ -1081,12 +1090,13 @@
// Only frame 0 has TPL block stats.
TplGopStats tpl_gop_stats;
- tpl_gop_stats.frame_stats_list.assign(3, { 8, 176, 144, {} });
+ tpl_gop_stats.frame_stats_list.assign(3, { 8, 176, 144, false, {} });
tpl_gop_stats.frame_stats_list[0] = CreateToyTplFrameStatsWithDiffSizes(8, 8);
AV1RateControlQMode rc;
const Status status =
- rc.GetGopEncodeInfo(gop_struct, tpl_gop_stats, RefFrameTable()).status();
+ rc.GetGopEncodeInfo(gop_struct, tpl_gop_stats, {}, RefFrameTable())
+ .status();
EXPECT_EQ(status.code, AOM_CODEC_INVALID_PARAM);
EXPECT_THAT(status.message,
HasSubstr("The frame with global_coding_idx 2 is a reference "
diff --git a/test/sad_test.cc b/test/sad_test.cc
index 55897f2..4c548d3 100644
--- a/test/sad_test.cc
+++ b/test/sad_test.cc
@@ -711,6 +711,7 @@
FillRandom(source_data_, source_stride_);
FillRandom(reference_data_, reference_stride_);
CheckSAD();
+ if (testing::Test::HasFatalFailure()) break;
test_count -= 1;
}
source_stride_ = tmp_stride;
@@ -765,6 +766,7 @@
FillRandom(source_data_, source_stride_);
FillRandom(reference_data_, reference_stride_);
CheckSAD();
+ if (testing::Test::HasFatalFailure()) break;
test_count -= 1;
}
source_stride_ = tmp_stride;
@@ -823,6 +825,7 @@
FillRandom(reference_data_, reference_stride_);
FillRandom(second_pred_, width_);
CheckSAD();
+ if (testing::Test::HasFatalFailure()) break;
test_count -= 1;
}
source_stride_ = tmp_stride;
@@ -900,6 +903,7 @@
FillRandom(source_data_, source_stride_);
FillRandom(reference_data_, reference_stride_);
CheckSAD();
+ if (testing::Test::HasFatalFailure()) break;
test_count -= 1;
}
source_stride_ = tmp_stride;
@@ -949,6 +953,7 @@
FillRandom(reference_data_, reference_stride_);
FillRandom(second_pred_, width_);
CheckSAD();
+ if (testing::Test::HasFatalFailure()) break;
test_count -= 1;
}
source_stride_ = tmp_stride;
@@ -1790,20 +1795,57 @@
#if HAVE_NEON
const SadMxNParam neon_tests[] = {
make_tuple(128, 128, &aom_sad128x128_neon, -1),
+ make_tuple(128, 64, &aom_sad128x64_neon, -1),
+ make_tuple(64, 128, &aom_sad64x128_neon, -1),
make_tuple(64, 64, &aom_sad64x64_neon, -1),
+ make_tuple(64, 32, &aom_sad64x32_neon, -1),
+ make_tuple(32, 64, &aom_sad32x64_neon, -1),
make_tuple(32, 32, &aom_sad32x32_neon, -1),
+ make_tuple(32, 16, &aom_sad32x16_neon, -1),
+ make_tuple(16, 32, &aom_sad16x32_neon, -1),
make_tuple(16, 16, &aom_sad16x16_neon, -1),
make_tuple(16, 8, &aom_sad16x8_neon, -1),
make_tuple(8, 16, &aom_sad8x16_neon, -1),
make_tuple(8, 8, &aom_sad8x8_neon, -1),
+ make_tuple(8, 4, &aom_sad8x4_neon, -1),
+ make_tuple(4, 8, &aom_sad4x8_neon, -1),
make_tuple(4, 4, &aom_sad4x4_neon, -1),
+#if !CONFIG_REALTIME_ONLY
+ make_tuple(64, 16, &aom_sad64x16_neon, -1),
+ make_tuple(32, 8, &aom_sad32x8_neon, -1),
+ make_tuple(16, 64, &aom_sad16x64_neon, -1),
+ make_tuple(16, 4, &aom_sad16x4_neon, -1),
+ make_tuple(8, 32, &aom_sad8x32_neon, -1),
+ make_tuple(4, 16, &aom_sad4x16_neon, -1),
+#endif
};
INSTANTIATE_TEST_SUITE_P(NEON, SADTest, ::testing::ValuesIn(neon_tests));
const SadMxNx4Param x4d_neon_tests[] = {
+ make_tuple(128, 128, &aom_sad128x128x4d_neon, -1),
+ make_tuple(128, 64, &aom_sad128x64x4d_neon, -1),
+ make_tuple(64, 128, &aom_sad64x128x4d_neon, -1),
make_tuple(64, 64, &aom_sad64x64x4d_neon, -1),
+ make_tuple(64, 32, &aom_sad64x32x4d_neon, -1),
+ make_tuple(32, 64, &aom_sad32x64x4d_neon, -1),
make_tuple(32, 32, &aom_sad32x32x4d_neon, -1),
+ make_tuple(32, 16, &aom_sad32x16x4d_neon, -1),
+ make_tuple(16, 32, &aom_sad16x32x4d_neon, -1),
make_tuple(16, 16, &aom_sad16x16x4d_neon, -1),
+ make_tuple(16, 8, &aom_sad16x8x4d_neon, -1),
+ make_tuple(8, 16, &aom_sad8x16x4d_neon, -1),
+ make_tuple(8, 8, &aom_sad8x8x4d_neon, -1),
+ make_tuple(8, 4, &aom_sad8x4x4d_neon, -1),
+ make_tuple(4, 8, &aom_sad4x8x4d_neon, -1),
+ make_tuple(4, 4, &aom_sad4x4x4d_neon, -1),
+#if !CONFIG_REALTIME_ONLY
+ make_tuple(64, 16, &aom_sad64x16x4d_neon, -1),
+ make_tuple(32, 8, &aom_sad32x8x4d_neon, -1),
+ make_tuple(16, 64, &aom_sad16x64x4d_neon, -1),
+ make_tuple(16, 4, &aom_sad16x4x4d_neon, -1),
+ make_tuple(8, 32, &aom_sad8x32x4d_neon, -1),
+ make_tuple(4, 16, &aom_sad4x16x4d_neon, -1),
+#endif
};
INSTANTIATE_TEST_SUITE_P(NEON, SADx4Test, ::testing::ValuesIn(x4d_neon_tests));
const SadSkipMxNParam skip_neon_tests[] = {
diff --git a/test/sse_sum_test.cc b/test/sse_sum_test.cc
index e7c32e6..68355ec 100644
--- a/test/sse_sum_test.cc
+++ b/test/sse_sum_test.cc
@@ -160,6 +160,13 @@
&aom_sum_sse_2d_i16_c, &aom_sum_sse_2d_i16_sse2)));
#endif // HAVE_SSE2
+
+#if HAVE_NEON
+INSTANTIATE_TEST_SUITE_P(NEON, SumSSETest,
+ ::testing::Values(TestFuncs(
+ &aom_sum_sse_2d_i16_c, &aom_sum_sse_2d_i16_neon)));
+#endif // HAVE_NEON
+
#if HAVE_AVX2
INSTANTIATE_TEST_SUITE_P(AVX2, SumSSETest,
::testing::Values(TestFuncs(
diff --git a/test/svc_datarate_test.cc b/test/svc_datarate_test.cc
index 770777c..af5d6d4 100644
--- a/test/svc_datarate_test.cc
+++ b/test/svc_datarate_test.cc
@@ -122,9 +122,9 @@
if (number_spatial_layers_ == 2) {
spatial_layer_id = (layer_frame_cnt_ % 2 == 0) ? 0 : 1;
} else if (number_spatial_layers_ == 3) {
- spatial_layer_id = (layer_frame_cnt_ % 3 == 0)
- ? 0
- : ((layer_frame_cnt_ - 1) % 3 == 0) ? 1 : 2;
+ spatial_layer_id = (layer_frame_cnt_ % 3 == 0) ? 0
+ : ((layer_frame_cnt_ - 1) % 3 == 0) ? 1
+ : 2;
}
// Set the reference/update flags, layer_id, and reference_map
// buffer index.
@@ -213,7 +213,7 @@
}
}
- virtual void EndPassHook(void) {
+ virtual void EndPassHook() {
duration_ = ((last_pts_ + 1) * timebase_);
for (int i = 0; i < number_temporal_layers_ * number_spatial_layers_; i++) {
effective_datarate_tl[i] = (effective_datarate_tl[i] / 1000) / duration_;
diff --git a/test/temporal_filter_test.cc b/test/temporal_filter_test.cc
index bf61f02..154fd5d 100644
--- a/test/temporal_filter_test.cc
+++ b/test/temporal_filter_test.cc
@@ -296,6 +296,15 @@
Combine(ValuesIn(temporal_filter_test_sse2),
Range(64, 65, 4)));
#endif // HAVE_SSE2
+
+#if HAVE_NEON
+TemporalFilterFuncParam temporal_filter_test_neon[] = { TemporalFilterFuncParam(
+ &av1_apply_temporal_filter_c, &av1_apply_temporal_filter_neon) };
+INSTANTIATE_TEST_SUITE_P(NEON, TemporalFilterTest,
+ Combine(ValuesIn(temporal_filter_test_neon),
+ Range(64, 65, 4)));
+#endif // HAVE_NEON
+
#if CONFIG_AV1_HIGHBITDEPTH
typedef void (*HBDTemporalFilterFunc)(
diff --git a/test/test-data.sha1 b/test/test-data.sha1
index 16673ab..a3074ce 100644
--- a/test/test-data.sha1
+++ b/test/test-data.sha1
@@ -566,3 +566,5 @@
e24aa6951afd7b2bb53eb1a73e25a19e7b189f82 *av1-1-b10-24-monochrome.ivf.md5
df0c9481104aa8c81f9e3b61b6d147a331ad3e35 *firstpass_stats
3eaf216d9fc8b4b9bb8c3956311f49a85974806c *bus_352x288_420_f20_b8.yuv
+c7f336958e7af6162c20ddc84d67c7dfa9826910 *av1-1-b8-16-intra_only-intrabc-extreme-dv.ivf
+36a4fcf07e645ed522cde5845dd9c6ab2b2d1502 *av1-1-b8-16-intra_only-intrabc-extreme-dv.ivf.md5
diff --git a/test/test.cmake b/test/test.cmake
index 9ad832e..ea99a3a 100644
--- a/test/test.cmake
+++ b/test/test.cmake
@@ -64,6 +64,8 @@
"${AOM_ROOT}/test/encode_test_driver.cc"
"${AOM_ROOT}/test/encode_test_driver.h"
"${AOM_ROOT}/test/end_to_end_psnr_test.cc"
+ "${AOM_ROOT}/test/forced_max_frame_width_height_test.cc"
+ "${AOM_ROOT}/test/force_key_frame_test.cc"
"${AOM_ROOT}/test/gf_pyr_height_test.cc"
"${AOM_ROOT}/test/rt_end_to_end_test.cc"
"${AOM_ROOT}/test/allintra_end_to_end_test.cc"
@@ -108,6 +110,7 @@
"${AOM_ROOT}/test/cpu_speed_test.cc"
"${AOM_ROOT}/test/cpu_used_firstpass_test.cc"
"${AOM_ROOT}/test/end_to_end_psnr_test.cc"
+ "${AOM_ROOT}/test/force_key_frame_test.cc"
"${AOM_ROOT}/test/gf_pyr_height_test.cc"
"${AOM_ROOT}/test/horz_superres_test.cc"
"${AOM_ROOT}/test/level_test.cc"
@@ -319,7 +322,7 @@
endif()
- if(HAVE_SSE4_2)
+ if(HAVE_SSE4_2 OR HAVE_ARM_CRC32)
list(APPEND AOM_UNIT_TEST_ENCODER_SOURCES "${AOM_ROOT}/test/hash_test.cc")
endif()
@@ -495,6 +498,10 @@
add_intrinsics_source_to_target("${AOM_NEON_INTRIN_FLAG}" "test_libaom"
"AOM_UNIT_TEST_COMMON_INTRIN_NEON")
endif()
+ if(HAVE_ARM_CRC32)
+ add_intrinsics_source_to_target("${AOM_ARM_CRC32_FLAG}" "test_libaom"
+ "AOM_UNIT_TEST_COMMON_INTRIN_CRC32")
+ endif()
if(ENABLE_TESTDATA)
make_test_data_lists("${AOM_UNIT_TEST_DATA_LIST_FILE}" test_files
diff --git a/test/test_data_util.cmake b/test/test_data_util.cmake
index 74154b6..a59ff45 100644
--- a/test/test_data_util.cmake
+++ b/test/test_data_util.cmake
@@ -518,6 +518,8 @@
"av1-1-b8-05-mv.ivf.md5"
"av1-1-b8-06-mfmv.ivf"
"av1-1-b8-06-mfmv.ivf.md5"
+ "av1-1-b8-16-intra_only-intrabc-extreme-dv.ivf"
+ "av1-1-b8-16-intra_only-intrabc-extreme-dv.ivf.md5"
"av1-1-b8-22-svc-L2T1.ivf"
"av1-1-b8-22-svc-L2T1.ivf.md5"
"av1-1-b8-22-svc-L1T2.ivf"
diff --git a/test/test_vectors.cc b/test/test_vectors.cc
index c38461e..09736d1 100644
--- a/test/test_vectors.cc
+++ b/test/test_vectors.cc
@@ -16,249 +16,252 @@
#define NELEMENTS(x) static_cast<int>(sizeof(x) / sizeof(x[0]))
#if CONFIG_AV1_DECODER
-const char *const kAV1TestVectors[] = { "av1-1-b8-00-quantizer-00.ivf",
- "av1-1-b8-00-quantizer-01.ivf",
- "av1-1-b8-00-quantizer-02.ivf",
- "av1-1-b8-00-quantizer-03.ivf",
- "av1-1-b8-00-quantizer-04.ivf",
- "av1-1-b8-00-quantizer-05.ivf",
- "av1-1-b8-00-quantizer-06.ivf",
- "av1-1-b8-00-quantizer-07.ivf",
- "av1-1-b8-00-quantizer-08.ivf",
- "av1-1-b8-00-quantizer-09.ivf",
- "av1-1-b8-00-quantizer-10.ivf",
- "av1-1-b8-00-quantizer-11.ivf",
- "av1-1-b8-00-quantizer-12.ivf",
- "av1-1-b8-00-quantizer-13.ivf",
- "av1-1-b8-00-quantizer-14.ivf",
- "av1-1-b8-00-quantizer-15.ivf",
- "av1-1-b8-00-quantizer-16.ivf",
- "av1-1-b8-00-quantizer-17.ivf",
- "av1-1-b8-00-quantizer-18.ivf",
- "av1-1-b8-00-quantizer-19.ivf",
- "av1-1-b8-00-quantizer-20.ivf",
- "av1-1-b8-00-quantizer-21.ivf",
- "av1-1-b8-00-quantizer-22.ivf",
- "av1-1-b8-00-quantizer-23.ivf",
- "av1-1-b8-00-quantizer-24.ivf",
- "av1-1-b8-00-quantizer-25.ivf",
- "av1-1-b8-00-quantizer-26.ivf",
- "av1-1-b8-00-quantizer-27.ivf",
- "av1-1-b8-00-quantizer-28.ivf",
- "av1-1-b8-00-quantizer-29.ivf",
- "av1-1-b8-00-quantizer-30.ivf",
- "av1-1-b8-00-quantizer-31.ivf",
- "av1-1-b8-00-quantizer-32.ivf",
- "av1-1-b8-00-quantizer-33.ivf",
- "av1-1-b8-00-quantizer-34.ivf",
- "av1-1-b8-00-quantizer-35.ivf",
- "av1-1-b8-00-quantizer-36.ivf",
- "av1-1-b8-00-quantizer-37.ivf",
- "av1-1-b8-00-quantizer-38.ivf",
- "av1-1-b8-00-quantizer-39.ivf",
- "av1-1-b8-00-quantizer-40.ivf",
- "av1-1-b8-00-quantizer-41.ivf",
- "av1-1-b8-00-quantizer-42.ivf",
- "av1-1-b8-00-quantizer-43.ivf",
- "av1-1-b8-00-quantizer-44.ivf",
- "av1-1-b8-00-quantizer-45.ivf",
- "av1-1-b8-00-quantizer-46.ivf",
- "av1-1-b8-00-quantizer-47.ivf",
- "av1-1-b8-00-quantizer-48.ivf",
- "av1-1-b8-00-quantizer-49.ivf",
- "av1-1-b8-00-quantizer-50.ivf",
- "av1-1-b8-00-quantizer-51.ivf",
- "av1-1-b8-00-quantizer-52.ivf",
- "av1-1-b8-00-quantizer-53.ivf",
- "av1-1-b8-00-quantizer-54.ivf",
- "av1-1-b8-00-quantizer-55.ivf",
- "av1-1-b8-00-quantizer-56.ivf",
- "av1-1-b8-00-quantizer-57.ivf",
- "av1-1-b8-00-quantizer-58.ivf",
- "av1-1-b8-00-quantizer-59.ivf",
- "av1-1-b8-00-quantizer-60.ivf",
- "av1-1-b8-00-quantizer-61.ivf",
- "av1-1-b8-00-quantizer-62.ivf",
- "av1-1-b8-00-quantizer-63.ivf",
+const char *const kAV1TestVectors[] = {
+ "av1-1-b8-00-quantizer-00.ivf",
+ "av1-1-b8-00-quantizer-01.ivf",
+ "av1-1-b8-00-quantizer-02.ivf",
+ "av1-1-b8-00-quantizer-03.ivf",
+ "av1-1-b8-00-quantizer-04.ivf",
+ "av1-1-b8-00-quantizer-05.ivf",
+ "av1-1-b8-00-quantizer-06.ivf",
+ "av1-1-b8-00-quantizer-07.ivf",
+ "av1-1-b8-00-quantizer-08.ivf",
+ "av1-1-b8-00-quantizer-09.ivf",
+ "av1-1-b8-00-quantizer-10.ivf",
+ "av1-1-b8-00-quantizer-11.ivf",
+ "av1-1-b8-00-quantizer-12.ivf",
+ "av1-1-b8-00-quantizer-13.ivf",
+ "av1-1-b8-00-quantizer-14.ivf",
+ "av1-1-b8-00-quantizer-15.ivf",
+ "av1-1-b8-00-quantizer-16.ivf",
+ "av1-1-b8-00-quantizer-17.ivf",
+ "av1-1-b8-00-quantizer-18.ivf",
+ "av1-1-b8-00-quantizer-19.ivf",
+ "av1-1-b8-00-quantizer-20.ivf",
+ "av1-1-b8-00-quantizer-21.ivf",
+ "av1-1-b8-00-quantizer-22.ivf",
+ "av1-1-b8-00-quantizer-23.ivf",
+ "av1-1-b8-00-quantizer-24.ivf",
+ "av1-1-b8-00-quantizer-25.ivf",
+ "av1-1-b8-00-quantizer-26.ivf",
+ "av1-1-b8-00-quantizer-27.ivf",
+ "av1-1-b8-00-quantizer-28.ivf",
+ "av1-1-b8-00-quantizer-29.ivf",
+ "av1-1-b8-00-quantizer-30.ivf",
+ "av1-1-b8-00-quantizer-31.ivf",
+ "av1-1-b8-00-quantizer-32.ivf",
+ "av1-1-b8-00-quantizer-33.ivf",
+ "av1-1-b8-00-quantizer-34.ivf",
+ "av1-1-b8-00-quantizer-35.ivf",
+ "av1-1-b8-00-quantizer-36.ivf",
+ "av1-1-b8-00-quantizer-37.ivf",
+ "av1-1-b8-00-quantizer-38.ivf",
+ "av1-1-b8-00-quantizer-39.ivf",
+ "av1-1-b8-00-quantizer-40.ivf",
+ "av1-1-b8-00-quantizer-41.ivf",
+ "av1-1-b8-00-quantizer-42.ivf",
+ "av1-1-b8-00-quantizer-43.ivf",
+ "av1-1-b8-00-quantizer-44.ivf",
+ "av1-1-b8-00-quantizer-45.ivf",
+ "av1-1-b8-00-quantizer-46.ivf",
+ "av1-1-b8-00-quantizer-47.ivf",
+ "av1-1-b8-00-quantizer-48.ivf",
+ "av1-1-b8-00-quantizer-49.ivf",
+ "av1-1-b8-00-quantizer-50.ivf",
+ "av1-1-b8-00-quantizer-51.ivf",
+ "av1-1-b8-00-quantizer-52.ivf",
+ "av1-1-b8-00-quantizer-53.ivf",
+ "av1-1-b8-00-quantizer-54.ivf",
+ "av1-1-b8-00-quantizer-55.ivf",
+ "av1-1-b8-00-quantizer-56.ivf",
+ "av1-1-b8-00-quantizer-57.ivf",
+ "av1-1-b8-00-quantizer-58.ivf",
+ "av1-1-b8-00-quantizer-59.ivf",
+ "av1-1-b8-00-quantizer-60.ivf",
+ "av1-1-b8-00-quantizer-61.ivf",
+ "av1-1-b8-00-quantizer-62.ivf",
+ "av1-1-b8-00-quantizer-63.ivf",
#if CONFIG_AV1_HIGHBITDEPTH
- "av1-1-b10-00-quantizer-00.ivf",
- "av1-1-b10-00-quantizer-01.ivf",
- "av1-1-b10-00-quantizer-02.ivf",
- "av1-1-b10-00-quantizer-03.ivf",
- "av1-1-b10-00-quantizer-04.ivf",
- "av1-1-b10-00-quantizer-05.ivf",
- "av1-1-b10-00-quantizer-06.ivf",
- "av1-1-b10-00-quantizer-07.ivf",
- "av1-1-b10-00-quantizer-08.ivf",
- "av1-1-b10-00-quantizer-09.ivf",
- "av1-1-b10-00-quantizer-10.ivf",
- "av1-1-b10-00-quantizer-11.ivf",
- "av1-1-b10-00-quantizer-12.ivf",
- "av1-1-b10-00-quantizer-13.ivf",
- "av1-1-b10-00-quantizer-14.ivf",
- "av1-1-b10-00-quantizer-15.ivf",
- "av1-1-b10-00-quantizer-16.ivf",
- "av1-1-b10-00-quantizer-17.ivf",
- "av1-1-b10-00-quantizer-18.ivf",
- "av1-1-b10-00-quantizer-19.ivf",
- "av1-1-b10-00-quantizer-20.ivf",
- "av1-1-b10-00-quantizer-21.ivf",
- "av1-1-b10-00-quantizer-22.ivf",
- "av1-1-b10-00-quantizer-23.ivf",
- "av1-1-b10-00-quantizer-24.ivf",
- "av1-1-b10-00-quantizer-25.ivf",
- "av1-1-b10-00-quantizer-26.ivf",
- "av1-1-b10-00-quantizer-27.ivf",
- "av1-1-b10-00-quantizer-28.ivf",
- "av1-1-b10-00-quantizer-29.ivf",
- "av1-1-b10-00-quantizer-30.ivf",
- "av1-1-b10-00-quantizer-31.ivf",
- "av1-1-b10-00-quantizer-32.ivf",
- "av1-1-b10-00-quantizer-33.ivf",
- "av1-1-b10-00-quantizer-34.ivf",
- "av1-1-b10-00-quantizer-35.ivf",
- "av1-1-b10-00-quantizer-36.ivf",
- "av1-1-b10-00-quantizer-37.ivf",
- "av1-1-b10-00-quantizer-38.ivf",
- "av1-1-b10-00-quantizer-39.ivf",
- "av1-1-b10-00-quantizer-40.ivf",
- "av1-1-b10-00-quantizer-41.ivf",
- "av1-1-b10-00-quantizer-42.ivf",
- "av1-1-b10-00-quantizer-43.ivf",
- "av1-1-b10-00-quantizer-44.ivf",
- "av1-1-b10-00-quantizer-45.ivf",
- "av1-1-b10-00-quantizer-46.ivf",
- "av1-1-b10-00-quantizer-47.ivf",
- "av1-1-b10-00-quantizer-48.ivf",
- "av1-1-b10-00-quantizer-49.ivf",
- "av1-1-b10-00-quantizer-50.ivf",
- "av1-1-b10-00-quantizer-51.ivf",
- "av1-1-b10-00-quantizer-52.ivf",
- "av1-1-b10-00-quantizer-53.ivf",
- "av1-1-b10-00-quantizer-54.ivf",
- "av1-1-b10-00-quantizer-55.ivf",
- "av1-1-b10-00-quantizer-56.ivf",
- "av1-1-b10-00-quantizer-57.ivf",
- "av1-1-b10-00-quantizer-58.ivf",
- "av1-1-b10-00-quantizer-59.ivf",
- "av1-1-b10-00-quantizer-60.ivf",
- "av1-1-b10-00-quantizer-61.ivf",
- "av1-1-b10-00-quantizer-62.ivf",
- "av1-1-b10-00-quantizer-63.ivf",
- "av1-1-b10-23-film_grain-50.ivf",
- "av1-1-b10-24-monochrome.ivf",
+ "av1-1-b10-00-quantizer-00.ivf",
+ "av1-1-b10-00-quantizer-01.ivf",
+ "av1-1-b10-00-quantizer-02.ivf",
+ "av1-1-b10-00-quantizer-03.ivf",
+ "av1-1-b10-00-quantizer-04.ivf",
+ "av1-1-b10-00-quantizer-05.ivf",
+ "av1-1-b10-00-quantizer-06.ivf",
+ "av1-1-b10-00-quantizer-07.ivf",
+ "av1-1-b10-00-quantizer-08.ivf",
+ "av1-1-b10-00-quantizer-09.ivf",
+ "av1-1-b10-00-quantizer-10.ivf",
+ "av1-1-b10-00-quantizer-11.ivf",
+ "av1-1-b10-00-quantizer-12.ivf",
+ "av1-1-b10-00-quantizer-13.ivf",
+ "av1-1-b10-00-quantizer-14.ivf",
+ "av1-1-b10-00-quantizer-15.ivf",
+ "av1-1-b10-00-quantizer-16.ivf",
+ "av1-1-b10-00-quantizer-17.ivf",
+ "av1-1-b10-00-quantizer-18.ivf",
+ "av1-1-b10-00-quantizer-19.ivf",
+ "av1-1-b10-00-quantizer-20.ivf",
+ "av1-1-b10-00-quantizer-21.ivf",
+ "av1-1-b10-00-quantizer-22.ivf",
+ "av1-1-b10-00-quantizer-23.ivf",
+ "av1-1-b10-00-quantizer-24.ivf",
+ "av1-1-b10-00-quantizer-25.ivf",
+ "av1-1-b10-00-quantizer-26.ivf",
+ "av1-1-b10-00-quantizer-27.ivf",
+ "av1-1-b10-00-quantizer-28.ivf",
+ "av1-1-b10-00-quantizer-29.ivf",
+ "av1-1-b10-00-quantizer-30.ivf",
+ "av1-1-b10-00-quantizer-31.ivf",
+ "av1-1-b10-00-quantizer-32.ivf",
+ "av1-1-b10-00-quantizer-33.ivf",
+ "av1-1-b10-00-quantizer-34.ivf",
+ "av1-1-b10-00-quantizer-35.ivf",
+ "av1-1-b10-00-quantizer-36.ivf",
+ "av1-1-b10-00-quantizer-37.ivf",
+ "av1-1-b10-00-quantizer-38.ivf",
+ "av1-1-b10-00-quantizer-39.ivf",
+ "av1-1-b10-00-quantizer-40.ivf",
+ "av1-1-b10-00-quantizer-41.ivf",
+ "av1-1-b10-00-quantizer-42.ivf",
+ "av1-1-b10-00-quantizer-43.ivf",
+ "av1-1-b10-00-quantizer-44.ivf",
+ "av1-1-b10-00-quantizer-45.ivf",
+ "av1-1-b10-00-quantizer-46.ivf",
+ "av1-1-b10-00-quantizer-47.ivf",
+ "av1-1-b10-00-quantizer-48.ivf",
+ "av1-1-b10-00-quantizer-49.ivf",
+ "av1-1-b10-00-quantizer-50.ivf",
+ "av1-1-b10-00-quantizer-51.ivf",
+ "av1-1-b10-00-quantizer-52.ivf",
+ "av1-1-b10-00-quantizer-53.ivf",
+ "av1-1-b10-00-quantizer-54.ivf",
+ "av1-1-b10-00-quantizer-55.ivf",
+ "av1-1-b10-00-quantizer-56.ivf",
+ "av1-1-b10-00-quantizer-57.ivf",
+ "av1-1-b10-00-quantizer-58.ivf",
+ "av1-1-b10-00-quantizer-59.ivf",
+ "av1-1-b10-00-quantizer-60.ivf",
+ "av1-1-b10-00-quantizer-61.ivf",
+ "av1-1-b10-00-quantizer-62.ivf",
+ "av1-1-b10-00-quantizer-63.ivf",
+ "av1-1-b10-23-film_grain-50.ivf",
+ "av1-1-b10-24-monochrome.ivf",
#endif // CONFIG_AV1_HIGHBITDEPTH
- "av1-1-b8-01-size-16x16.ivf",
- "av1-1-b8-01-size-16x18.ivf",
- "av1-1-b8-01-size-16x32.ivf",
- "av1-1-b8-01-size-16x34.ivf",
- "av1-1-b8-01-size-16x64.ivf",
- "av1-1-b8-01-size-16x66.ivf",
- "av1-1-b8-01-size-18x16.ivf",
- "av1-1-b8-01-size-18x18.ivf",
- "av1-1-b8-01-size-18x32.ivf",
- "av1-1-b8-01-size-18x34.ivf",
- "av1-1-b8-01-size-18x64.ivf",
- "av1-1-b8-01-size-18x66.ivf",
- "av1-1-b8-01-size-196x196.ivf",
- "av1-1-b8-01-size-196x198.ivf",
- "av1-1-b8-01-size-196x200.ivf",
- "av1-1-b8-01-size-196x202.ivf",
- "av1-1-b8-01-size-196x208.ivf",
- "av1-1-b8-01-size-196x210.ivf",
- "av1-1-b8-01-size-196x224.ivf",
- "av1-1-b8-01-size-196x226.ivf",
- "av1-1-b8-01-size-198x196.ivf",
- "av1-1-b8-01-size-198x198.ivf",
- "av1-1-b8-01-size-198x200.ivf",
- "av1-1-b8-01-size-198x202.ivf",
- "av1-1-b8-01-size-198x208.ivf",
- "av1-1-b8-01-size-198x210.ivf",
- "av1-1-b8-01-size-198x224.ivf",
- "av1-1-b8-01-size-198x226.ivf",
- "av1-1-b8-01-size-200x196.ivf",
- "av1-1-b8-01-size-200x198.ivf",
- "av1-1-b8-01-size-200x200.ivf",
- "av1-1-b8-01-size-200x202.ivf",
- "av1-1-b8-01-size-200x208.ivf",
- "av1-1-b8-01-size-200x210.ivf",
- "av1-1-b8-01-size-200x224.ivf",
- "av1-1-b8-01-size-200x226.ivf",
- "av1-1-b8-01-size-202x196.ivf",
- "av1-1-b8-01-size-202x198.ivf",
- "av1-1-b8-01-size-202x200.ivf",
- "av1-1-b8-01-size-202x202.ivf",
- "av1-1-b8-01-size-202x208.ivf",
- "av1-1-b8-01-size-202x210.ivf",
- "av1-1-b8-01-size-202x224.ivf",
- "av1-1-b8-01-size-202x226.ivf",
- "av1-1-b8-01-size-208x196.ivf",
- "av1-1-b8-01-size-208x198.ivf",
- "av1-1-b8-01-size-208x200.ivf",
- "av1-1-b8-01-size-208x202.ivf",
- "av1-1-b8-01-size-208x208.ivf",
- "av1-1-b8-01-size-208x210.ivf",
- "av1-1-b8-01-size-208x224.ivf",
- "av1-1-b8-01-size-208x226.ivf",
- "av1-1-b8-01-size-210x196.ivf",
- "av1-1-b8-01-size-210x198.ivf",
- "av1-1-b8-01-size-210x200.ivf",
- "av1-1-b8-01-size-210x202.ivf",
- "av1-1-b8-01-size-210x208.ivf",
- "av1-1-b8-01-size-210x210.ivf",
- "av1-1-b8-01-size-210x224.ivf",
- "av1-1-b8-01-size-210x226.ivf",
- "av1-1-b8-01-size-224x196.ivf",
- "av1-1-b8-01-size-224x198.ivf",
- "av1-1-b8-01-size-224x200.ivf",
- "av1-1-b8-01-size-224x202.ivf",
- "av1-1-b8-01-size-224x208.ivf",
- "av1-1-b8-01-size-224x210.ivf",
- "av1-1-b8-01-size-224x224.ivf",
- "av1-1-b8-01-size-224x226.ivf",
- "av1-1-b8-01-size-226x196.ivf",
- "av1-1-b8-01-size-226x198.ivf",
- "av1-1-b8-01-size-226x200.ivf",
- "av1-1-b8-01-size-226x202.ivf",
- "av1-1-b8-01-size-226x208.ivf",
- "av1-1-b8-01-size-226x210.ivf",
- "av1-1-b8-01-size-226x224.ivf",
- "av1-1-b8-01-size-226x226.ivf",
- "av1-1-b8-01-size-32x16.ivf",
- "av1-1-b8-01-size-32x18.ivf",
- "av1-1-b8-01-size-32x32.ivf",
- "av1-1-b8-01-size-32x34.ivf",
- "av1-1-b8-01-size-32x64.ivf",
- "av1-1-b8-01-size-32x66.ivf",
- "av1-1-b8-01-size-34x16.ivf",
- "av1-1-b8-01-size-34x18.ivf",
- "av1-1-b8-01-size-34x32.ivf",
- "av1-1-b8-01-size-34x34.ivf",
- "av1-1-b8-01-size-34x64.ivf",
- "av1-1-b8-01-size-34x66.ivf",
- "av1-1-b8-01-size-64x16.ivf",
- "av1-1-b8-01-size-64x18.ivf",
- "av1-1-b8-01-size-64x32.ivf",
- "av1-1-b8-01-size-64x34.ivf",
- "av1-1-b8-01-size-64x64.ivf",
- "av1-1-b8-01-size-64x66.ivf",
- "av1-1-b8-01-size-66x16.ivf",
- "av1-1-b8-01-size-66x18.ivf",
- "av1-1-b8-01-size-66x32.ivf",
- "av1-1-b8-01-size-66x34.ivf",
- "av1-1-b8-01-size-66x64.ivf",
- "av1-1-b8-01-size-66x66.ivf",
- "av1-1-b8-02-allintra.ivf",
- "av1-1-b8-03-sizedown.mkv",
- "av1-1-b8-03-sizeup.mkv",
- "av1-1-b8-04-cdfupdate.ivf",
- "av1-1-b8-05-mv.ivf",
- "av1-1-b8-06-mfmv.ivf",
- "av1-1-b8-22-svc-L1T2.ivf",
- "av1-1-b8-22-svc-L2T1.ivf",
- "av1-1-b8-22-svc-L2T2.ivf",
- "av1-1-b8-23-film_grain-50.ivf",
- "av1-1-b8-24-monochrome.ivf" };
+ "av1-1-b8-01-size-16x16.ivf",
+ "av1-1-b8-01-size-16x18.ivf",
+ "av1-1-b8-01-size-16x32.ivf",
+ "av1-1-b8-01-size-16x34.ivf",
+ "av1-1-b8-01-size-16x64.ivf",
+ "av1-1-b8-01-size-16x66.ivf",
+ "av1-1-b8-01-size-18x16.ivf",
+ "av1-1-b8-01-size-18x18.ivf",
+ "av1-1-b8-01-size-18x32.ivf",
+ "av1-1-b8-01-size-18x34.ivf",
+ "av1-1-b8-01-size-18x64.ivf",
+ "av1-1-b8-01-size-18x66.ivf",
+ "av1-1-b8-01-size-196x196.ivf",
+ "av1-1-b8-01-size-196x198.ivf",
+ "av1-1-b8-01-size-196x200.ivf",
+ "av1-1-b8-01-size-196x202.ivf",
+ "av1-1-b8-01-size-196x208.ivf",
+ "av1-1-b8-01-size-196x210.ivf",
+ "av1-1-b8-01-size-196x224.ivf",
+ "av1-1-b8-01-size-196x226.ivf",
+ "av1-1-b8-01-size-198x196.ivf",
+ "av1-1-b8-01-size-198x198.ivf",
+ "av1-1-b8-01-size-198x200.ivf",
+ "av1-1-b8-01-size-198x202.ivf",
+ "av1-1-b8-01-size-198x208.ivf",
+ "av1-1-b8-01-size-198x210.ivf",
+ "av1-1-b8-01-size-198x224.ivf",
+ "av1-1-b8-01-size-198x226.ivf",
+ "av1-1-b8-01-size-200x196.ivf",
+ "av1-1-b8-01-size-200x198.ivf",
+ "av1-1-b8-01-size-200x200.ivf",
+ "av1-1-b8-01-size-200x202.ivf",
+ "av1-1-b8-01-size-200x208.ivf",
+ "av1-1-b8-01-size-200x210.ivf",
+ "av1-1-b8-01-size-200x224.ivf",
+ "av1-1-b8-01-size-200x226.ivf",
+ "av1-1-b8-01-size-202x196.ivf",
+ "av1-1-b8-01-size-202x198.ivf",
+ "av1-1-b8-01-size-202x200.ivf",
+ "av1-1-b8-01-size-202x202.ivf",
+ "av1-1-b8-01-size-202x208.ivf",
+ "av1-1-b8-01-size-202x210.ivf",
+ "av1-1-b8-01-size-202x224.ivf",
+ "av1-1-b8-01-size-202x226.ivf",
+ "av1-1-b8-01-size-208x196.ivf",
+ "av1-1-b8-01-size-208x198.ivf",
+ "av1-1-b8-01-size-208x200.ivf",
+ "av1-1-b8-01-size-208x202.ivf",
+ "av1-1-b8-01-size-208x208.ivf",
+ "av1-1-b8-01-size-208x210.ivf",
+ "av1-1-b8-01-size-208x224.ivf",
+ "av1-1-b8-01-size-208x226.ivf",
+ "av1-1-b8-01-size-210x196.ivf",
+ "av1-1-b8-01-size-210x198.ivf",
+ "av1-1-b8-01-size-210x200.ivf",
+ "av1-1-b8-01-size-210x202.ivf",
+ "av1-1-b8-01-size-210x208.ivf",
+ "av1-1-b8-01-size-210x210.ivf",
+ "av1-1-b8-01-size-210x224.ivf",
+ "av1-1-b8-01-size-210x226.ivf",
+ "av1-1-b8-01-size-224x196.ivf",
+ "av1-1-b8-01-size-224x198.ivf",
+ "av1-1-b8-01-size-224x200.ivf",
+ "av1-1-b8-01-size-224x202.ivf",
+ "av1-1-b8-01-size-224x208.ivf",
+ "av1-1-b8-01-size-224x210.ivf",
+ "av1-1-b8-01-size-224x224.ivf",
+ "av1-1-b8-01-size-224x226.ivf",
+ "av1-1-b8-01-size-226x196.ivf",
+ "av1-1-b8-01-size-226x198.ivf",
+ "av1-1-b8-01-size-226x200.ivf",
+ "av1-1-b8-01-size-226x202.ivf",
+ "av1-1-b8-01-size-226x208.ivf",
+ "av1-1-b8-01-size-226x210.ivf",
+ "av1-1-b8-01-size-226x224.ivf",
+ "av1-1-b8-01-size-226x226.ivf",
+ "av1-1-b8-01-size-32x16.ivf",
+ "av1-1-b8-01-size-32x18.ivf",
+ "av1-1-b8-01-size-32x32.ivf",
+ "av1-1-b8-01-size-32x34.ivf",
+ "av1-1-b8-01-size-32x64.ivf",
+ "av1-1-b8-01-size-32x66.ivf",
+ "av1-1-b8-01-size-34x16.ivf",
+ "av1-1-b8-01-size-34x18.ivf",
+ "av1-1-b8-01-size-34x32.ivf",
+ "av1-1-b8-01-size-34x34.ivf",
+ "av1-1-b8-01-size-34x64.ivf",
+ "av1-1-b8-01-size-34x66.ivf",
+ "av1-1-b8-01-size-64x16.ivf",
+ "av1-1-b8-01-size-64x18.ivf",
+ "av1-1-b8-01-size-64x32.ivf",
+ "av1-1-b8-01-size-64x34.ivf",
+ "av1-1-b8-01-size-64x64.ivf",
+ "av1-1-b8-01-size-64x66.ivf",
+ "av1-1-b8-01-size-66x16.ivf",
+ "av1-1-b8-01-size-66x18.ivf",
+ "av1-1-b8-01-size-66x32.ivf",
+ "av1-1-b8-01-size-66x34.ivf",
+ "av1-1-b8-01-size-66x64.ivf",
+ "av1-1-b8-01-size-66x66.ivf",
+ "av1-1-b8-02-allintra.ivf",
+ "av1-1-b8-03-sizedown.mkv",
+ "av1-1-b8-03-sizeup.mkv",
+ "av1-1-b8-04-cdfupdate.ivf",
+ "av1-1-b8-05-mv.ivf",
+ "av1-1-b8-06-mfmv.ivf",
+ "av1-1-b8-16-intra_only-intrabc-extreme-dv.ivf",
+ "av1-1-b8-22-svc-L1T2.ivf",
+ "av1-1-b8-22-svc-L2T1.ivf",
+ "av1-1-b8-22-svc-L2T2.ivf",
+ "av1-1-b8-23-film_grain-50.ivf",
+ "av1-1-b8-24-monochrome.ivf"
+};
const int kNumAV1TestVectors = NELEMENTS(kAV1TestVectors);
#endif // CONFIG_AV1_DECODER
diff --git a/test/variance_test.cc b/test/variance_test.cc
index e96f933..46b87b1 100644
--- a/test/variance_test.cc
+++ b/test/variance_test.cc
@@ -2819,28 +2819,41 @@
&aom_get4x4sse_cs_neon)));
INSTANTIATE_TEST_SUITE_P(NEON, AvxMseTest,
- ::testing::Values(MseParams(4, 4,
- &aom_mse16x16_neon)));
+ ::testing::Values(MseParams(3, 3, &aom_mse8x8_neon),
+ MseParams(3, 4, &aom_mse8x16_neon),
+ MseParams(4, 4, &aom_mse16x16_neon),
+ MseParams(4, 3, &aom_mse16x8_neon)));
-INSTANTIATE_TEST_SUITE_P(
- NEON, AvxVarianceTest,
- ::testing::Values(VarianceParams(7, 7, &aom_variance128x128_neon),
- VarianceParams(6, 6, &aom_variance64x64_neon),
- VarianceParams(7, 6, &aom_variance128x64_neon),
- VarianceParams(6, 7, &aom_variance64x128_neon),
- VarianceParams(6, 6, &aom_variance64x64_neon),
- VarianceParams(6, 5, &aom_variance64x32_neon),
- VarianceParams(5, 6, &aom_variance32x64_neon),
- VarianceParams(5, 5, &aom_variance32x32_neon),
- VarianceParams(5, 4, &aom_variance32x16_neon),
- VarianceParams(4, 5, &aom_variance16x32_neon),
- VarianceParams(4, 4, &aom_variance16x16_neon),
- VarianceParams(4, 3, &aom_variance16x8_neon),
- VarianceParams(3, 4, &aom_variance8x16_neon),
- VarianceParams(3, 3, &aom_variance8x8_neon),
- VarianceParams(3, 2, &aom_variance8x4_neon),
- VarianceParams(2, 3, &aom_variance4x8_neon),
- VarianceParams(2, 2, &aom_variance4x4_neon)));
+const VarianceParams kArrayVariance_neon[] = {
+ VarianceParams(7, 7, &aom_variance128x128_neon),
+ VarianceParams(6, 6, &aom_variance64x64_neon),
+ VarianceParams(7, 6, &aom_variance128x64_neon),
+ VarianceParams(6, 7, &aom_variance64x128_neon),
+ VarianceParams(6, 6, &aom_variance64x64_neon),
+ VarianceParams(6, 5, &aom_variance64x32_neon),
+ VarianceParams(5, 6, &aom_variance32x64_neon),
+ VarianceParams(5, 5, &aom_variance32x32_neon),
+ VarianceParams(5, 4, &aom_variance32x16_neon),
+ VarianceParams(4, 5, &aom_variance16x32_neon),
+ VarianceParams(4, 4, &aom_variance16x16_neon),
+ VarianceParams(4, 3, &aom_variance16x8_neon),
+ VarianceParams(3, 4, &aom_variance8x16_neon),
+ VarianceParams(3, 3, &aom_variance8x8_neon),
+ VarianceParams(3, 2, &aom_variance8x4_neon),
+ VarianceParams(2, 3, &aom_variance4x8_neon),
+ VarianceParams(2, 2, &aom_variance4x4_neon),
+#if !CONFIG_REALTIME_ONLY
+ VarianceParams(2, 4, &aom_variance4x16_neon),
+ VarianceParams(4, 2, &aom_variance16x4_neon),
+ VarianceParams(3, 5, &aom_variance8x32_neon),
+ VarianceParams(5, 3, &aom_variance32x8_neon),
+ VarianceParams(4, 6, &aom_variance16x64_neon),
+ VarianceParams(6, 4, &aom_variance64x16_neon),
+#endif
+};
+
+INSTANTIATE_TEST_SUITE_P(NEON, AvxVarianceTest,
+ ::testing::ValuesIn(kArrayVariance_neon));
const SubpelVarianceParams kArraySubpelVariance_neon[] = {
SubpelVarianceParams(7, 7, &aom_sub_pixel_variance128x128_neon, 0),
diff --git a/test/video_source.h b/test/video_source.h
index 742178e..b51cbf4 100644
--- a/test/video_source.h
+++ b/test/video_source.h
@@ -215,7 +215,7 @@
aom_img_free(img_);
img_ = aom_img_alloc(nullptr, format_, width_, height_, 32);
ASSERT_NE(img_, nullptr);
- raw_sz_ = ((img_->w + 31) & ~31) * img_->h * img_->bps / 8;
+ raw_sz_ = ((img_->w + 31) & ~31u) * img_->h * img_->bps / 8;
}
aom_image_t *img_;
diff --git a/test/wiener_test.cc b/test/wiener_test.cc
index 69df5ea..d44dd92 100644
--- a/test/wiener_test.cc
+++ b/test/wiener_test.cc
@@ -10,6 +10,7 @@
*/
#include <tuple>
+#include <utility>
#include <vector>
#include "third_party/googletest/src/googletest/include/gtest/gtest.h"
@@ -221,10 +222,12 @@
// will always be multiples of 64 when called from non-test code.
// If in future any new requirements are added, these lines will
// need changing.
- const int h_start = (rng_.Rand16() % (MAX_WIENER_BLOCK / 2)) & ~1;
+ int h_start = (rng_.Rand16() % (MAX_WIENER_BLOCK / 2)) & ~1;
int h_end = run_times != 1 ? 256 : (rng_.Rand16() % MAX_WIENER_BLOCK);
- const int v_start = rng_.Rand16() % (MAX_WIENER_BLOCK / 2);
+ if (h_start > h_end) std::swap(h_start, h_end);
+ int v_start = rng_.Rand16() % (MAX_WIENER_BLOCK / 2);
int v_end = run_times != 1 ? 256 : (rng_.Rand16() % MAX_WIENER_BLOCK);
+ if (v_start > v_end) std::swap(v_start, v_end);
const int dgd_stride = h_end;
const int src_stride = MAX_DATA_BLOCK;
const int iters = run_times == 1 ? kIterations : 2;
@@ -551,10 +554,12 @@
// will always be multiples of 64 when called from non-test code.
// If in future any new requirements are added, these lines will
// need changing.
- const int h_start = (rng_.Rand16() % (MAX_WIENER_BLOCK / 2)) & ~1;
+ int h_start = (rng_.Rand16() % (MAX_WIENER_BLOCK / 2)) & ~1;
int h_end = run_times != 1 ? 256 : (rng_.Rand16() % MAX_WIENER_BLOCK);
- const int v_start = rng_.Rand16() % (MAX_WIENER_BLOCK / 2);
+ if (h_start > h_end) std::swap(h_start, h_end);
+ int v_start = rng_.Rand16() % (MAX_WIENER_BLOCK / 2);
int v_end = run_times != 1 ? 256 : (rng_.Rand16() % MAX_WIENER_BLOCK);
+ if (v_start > v_end) std::swap(v_start, v_end);
const int dgd_stride = h_end;
const int src_stride = MAX_DATA_BLOCK;
const int iters = run_times == 1 ? kIterations : 2;
diff --git a/third_party/googletest/README.libaom b/third_party/googletest/README.libaom
index a461f36..5e429d4 100644
--- a/third_party/googletest/README.libaom
+++ b/third_party/googletest/README.libaom
@@ -1,5 +1,5 @@
URL: https://github.com/google/googletest
-Version: release-1.11.0
+Version: release-1.12.1
License: BSD
License File: LICENSE
@@ -13,6 +13,7 @@
Local Modifications:
- Remove everything but:
+ .clang-format
CMakeLists.txt
CONTRIBUTORS
googlemock/
@@ -29,3 +30,9 @@
src
LICENSE
README.md
+- In googletest/include/gtest/internal/custom/gtest-port.h, define
+ GTEST_HAS_NOTIFICATION_ as 1 and use a stub Notification class to fix
+ the mingw32 g++ compilation errors caused by the lack of std::mutex
+ and std::condition_variable in the <mutex> and <condition_variable>
+ headers if mingw32 is configured with the win32 threads option. See
+ https://stackoverflow.com/questions/17242516/mingw-w64-threads-posix-vs-win32
diff --git a/third_party/googletest/src/.clang-format b/third_party/googletest/src/.clang-format
new file mode 100644
index 0000000..5b9bfe6
--- /dev/null
+++ b/third_party/googletest/src/.clang-format
@@ -0,0 +1,4 @@
+# Run manually to reformat a file:
+# clang-format -i --style=file <file>
+Language: Cpp
+BasedOnStyle: Google
diff --git a/third_party/googletest/src/CMakeLists.txt b/third_party/googletest/src/CMakeLists.txt
index ea81ab1..102e28c 100644
--- a/third_party/googletest/src/CMakeLists.txt
+++ b/third_party/googletest/src/CMakeLists.txt
@@ -1,19 +1,21 @@
# Note: CMake support is community-based. The maintainers do not use CMake
# internally.
-cmake_minimum_required(VERSION 2.8.12)
+cmake_minimum_required(VERSION 3.5)
if (POLICY CMP0048)
cmake_policy(SET CMP0048 NEW)
endif (POLICY CMP0048)
-project(googletest-distribution)
-set(GOOGLETEST_VERSION 1.11.0)
+if (POLICY CMP0077)
+ cmake_policy(SET CMP0077 NEW)
+endif (POLICY CMP0077)
-if (CMAKE_VERSION VERSION_GREATER "3.0.2")
- if(NOT CYGWIN AND NOT MSYS AND NOT ${CMAKE_SYSTEM_NAME} STREQUAL QNX)
- set(CMAKE_CXX_EXTENSIONS OFF)
- endif()
+project(googletest-distribution)
+set(GOOGLETEST_VERSION 1.12.1)
+
+if(NOT CYGWIN AND NOT MSYS AND NOT ${CMAKE_SYSTEM_NAME} STREQUAL QNX)
+ set(CMAKE_CXX_EXTENSIONS OFF)
endif()
enable_testing()
diff --git a/third_party/googletest/src/CONTRIBUTORS b/third_party/googletest/src/CONTRIBUTORS
index 76db0b4..77397a5 100644
--- a/third_party/googletest/src/CONTRIBUTORS
+++ b/third_party/googletest/src/CONTRIBUTORS
@@ -34,6 +34,7 @@
Mario Tanev <radix@google.com>
Mark Paskin
Markus Heule <markus.heule@gmail.com>
+Martijn Vels <mvels@google.com>
Matthew Simmons <simmonmt@acm.org>
Mika Raento <mikie@iki.fi>
Mike Bland <mbland@google.com>
@@ -55,6 +56,7 @@
Sean Mcafee <eefacm@gmail.com>
Sigurður Ásgeirsson <siggi@google.com>
Sverre Sundsdal <sundsdal@gmail.com>
+Szymon Sobik <sobik.szymon@gmail.com>
Takeshi Yoshino <tyoshino@google.com>
Tracy Bialik <tracy@pentad.com>
Vadim Berman <vadimb@google.com>
diff --git a/third_party/googletest/src/README.md b/third_party/googletest/src/README.md
index 7d872a5..30edaec 100644
--- a/third_party/googletest/src/README.md
+++ b/third_party/googletest/src/README.md
@@ -6,7 +6,8 @@
GoogleTest now follows the
[Abseil Live at Head philosophy](https://abseil.io/about/philosophy#upgrade-support).
-We recommend using the latest commit in the `master` branch in your projects.
+We recommend
+[updating to the latest commit in the `main` branch as often as possible](https://github.com/abseil/abseil-cpp/blob/master/FAQ.md#what-is-live-at-head-and-how-do-i-do-it).
#### Documentation Updates
@@ -14,9 +15,9 @@
https://google.github.io/googletest/. We recommend browsing the documentation on
GitHub Pages rather than directly in the repository.
-#### Release 1.10.x
+#### Release 1.11.0
-[Release 1.10.x](https://github.com/google/googletest/releases/tag/release-1.10.0)
+[Release 1.11.0](https://github.com/google/googletest/releases/tag/release-1.11.0)
is now available.
#### Coming Soon
@@ -109,8 +110,8 @@
[GoogleTest UI](https://github.com/ospector/gtest-gbar) is a test runner that
runs your test binary, allows you to track its progress via a progress bar, and
-displays a list of test failures. Clicking on one shows failure text. Google
-Test UI is written in C#.
+displays a list of test failures. Clicking on one shows failure text. GoogleTest
+UI is written in C#.
[GTest TAP Listener](https://github.com/kinow/gtest-tap-listener) is an event
listener for GoogleTest that implements the
@@ -121,11 +122,11 @@
runs tests from your binary in parallel to provide significant speed-up.
[GoogleTest Adapter](https://marketplace.visualstudio.com/items?itemName=DavidSchuldenfrei.gtest-adapter)
-is a VS Code extension allowing to view GoogleTest in a tree view, and run/debug
+is a VS Code extension allowing to view GoogleTest in a tree view and run/debug
your tests.
[C++ TestMate](https://github.com/matepek/vscode-catch2-test-adapter) is a VS
-Code extension allowing to view GoogleTest in a tree view, and run/debug your
+Code extension allowing to view GoogleTest in a tree view and run/debug your
tests.
[Cornichon](https://pypi.org/project/cornichon/) is a small Gherkin DSL parser
diff --git a/third_party/googletest/src/googlemock/CMakeLists.txt b/third_party/googletest/src/googlemock/CMakeLists.txt
index e7df8ec..5c1f0da 100644
--- a/third_party/googletest/src/googlemock/CMakeLists.txt
+++ b/third_party/googletest/src/googlemock/CMakeLists.txt
@@ -36,13 +36,9 @@
# as ${gmock_SOURCE_DIR} and to the root binary directory as
# ${gmock_BINARY_DIR}.
# Language "C" is required for find_package(Threads).
-if (CMAKE_VERSION VERSION_LESS 3.0)
- project(gmock CXX C)
-else()
- cmake_policy(SET CMP0048 NEW)
- project(gmock VERSION ${GOOGLETEST_VERSION} LANGUAGES CXX C)
-endif()
-cmake_minimum_required(VERSION 2.8.12)
+cmake_minimum_required(VERSION 3.5)
+cmake_policy(SET CMP0048 NEW)
+project(gmock VERSION ${GOOGLETEST_VERSION} LANGUAGES CXX C)
if (COMMAND set_up_hermetic_build)
set_up_hermetic_build()
@@ -109,11 +105,12 @@
# to the targets for when we are part of a parent build (ie being pulled
# in via add_subdirectory() rather than being a standalone build).
if (DEFINED CMAKE_VERSION AND NOT "${CMAKE_VERSION}" VERSION_LESS "2.8.11")
+ string(REPLACE ";" "$<SEMICOLON>" dirs "${gmock_build_include_dirs}")
target_include_directories(gmock SYSTEM INTERFACE
- "$<BUILD_INTERFACE:${gmock_build_include_dirs}>"
+ "$<BUILD_INTERFACE:${dirs}>"
"$<INSTALL_INTERFACE:$<INSTALL_PREFIX>/${CMAKE_INSTALL_INCLUDEDIR}>")
target_include_directories(gmock_main SYSTEM INTERFACE
- "$<BUILD_INTERFACE:${gmock_build_include_dirs}>"
+ "$<BUILD_INTERFACE:${dirs}>"
"$<INSTALL_INTERFACE:$<INSTALL_PREFIX>/${CMAKE_INSTALL_INCLUDEDIR}>")
endif()
@@ -154,7 +151,10 @@
cxx_test(gmock_ex_test gmock_main)
cxx_test(gmock-function-mocker_test gmock_main)
cxx_test(gmock-internal-utils_test gmock_main)
- cxx_test(gmock-matchers_test gmock_main)
+ cxx_test(gmock-matchers-arithmetic_test gmock_main)
+ cxx_test(gmock-matchers-comparisons_test gmock_main)
+ cxx_test(gmock-matchers-containers_test gmock_main)
+ cxx_test(gmock-matchers-misc_test gmock_main)
cxx_test(gmock-more-actions_test gmock_main)
cxx_test(gmock-nice-strict_test gmock_main)
cxx_test(gmock-port_test gmock_main)
diff --git a/third_party/googletest/src/googlemock/README.md b/third_party/googletest/src/googlemock/README.md
index ead6883..7da6065 100644
--- a/third_party/googletest/src/googlemock/README.md
+++ b/third_party/googletest/src/googlemock/README.md
@@ -35,10 +35,6 @@
* [gMock Cookbook](https://google.github.io/googletest/gmock_cook_book.html)
* [gMock Cheat Sheet](https://google.github.io/googletest/gmock_cheat_sheet.html)
-Please note that code under scripts/generator/ is from the
-[cppclean project](http://code.google.com/p/cppclean/) and under the Apache
-License, which is different from GoogleMock's license.
-
GoogleMock is a part of
[GoogleTest C++ testing framework](http://github.com/google/googletest/) and a
subject to the same requirements.
diff --git a/third_party/googletest/src/googlemock/include/gmock/gmock-actions.h b/third_party/googletest/src/googlemock/include/gmock/gmock-actions.h
index f2393bd..c785ad8 100644
--- a/third_party/googletest/src/googlemock/include/gmock/gmock-actions.h
+++ b/third_party/googletest/src/googlemock/include/gmock/gmock-actions.h
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Google Mock - a framework for writing C++ mock classes.
//
// The ACTION* family of macros can be used in a namespace scope to
@@ -125,13 +124,14 @@
// To learn more about using these macros, please search for 'ACTION' on
// https://github.com/google/googletest/blob/master/docs/gmock_cook_book.md
-// GOOGLETEST_CM0002 DO NOT DELETE
+// IWYU pragma: private, include "gmock/gmock.h"
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_ACTIONS_H_
#define GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_ACTIONS_H_
#ifndef _WIN32_WCE
-# include <errno.h>
+#include <errno.h>
#endif
#include <algorithm>
@@ -147,8 +147,8 @@
#include "gmock/internal/gmock-pp.h"
#ifdef _MSC_VER
-# pragma warning(push)
-# pragma warning(disable:4100)
+#pragma warning(push)
+#pragma warning(disable : 4100)
#endif
namespace testing {
@@ -196,9 +196,7 @@
public:
// This function returns true if and only if type T has a built-in default
// value.
- static bool Exists() {
- return ::std::is_default_constructible<T>::value;
- }
+ static bool Exists() { return ::std::is_default_constructible<T>::value; }
static T Get() {
return BuiltInDefaultValueGetter<
@@ -227,11 +225,11 @@
// The following specializations define the default values for
// specific types we care about.
#define GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_(type, value) \
- template <> \
- class BuiltInDefaultValue<type> { \
- public: \
- static bool Exists() { return true; } \
- static type Get() { return value; } \
+ template <> \
+ class BuiltInDefaultValue<type> { \
+ public: \
+ static bool Exists() { return true; } \
+ static type Get() { return value; } \
}
GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_(void, ); // NOLINT
@@ -255,21 +253,309 @@
GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_(signed short, 0); // NOLINT
GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_(unsigned int, 0U);
GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_(signed int, 0);
-GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_(unsigned long, 0UL); // NOLINT
-GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_(signed long, 0L); // NOLINT
+GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_(unsigned long, 0UL); // NOLINT
+GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_(signed long, 0L); // NOLINT
GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_(unsigned long long, 0); // NOLINT
-GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_(signed long long, 0); // NOLINT
+GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_(signed long long, 0); // NOLINT
GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_(float, 0);
GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_(double, 0);
#undef GMOCK_DEFINE_DEFAULT_ACTION_FOR_RETURN_TYPE_
-// Simple two-arg form of std::disjunction.
-template <typename P, typename Q>
-using disjunction = typename ::std::conditional<P::value, P, Q>::type;
+// Partial implementations of metaprogramming types from the standard library
+// not available in C++11.
+
+template <typename P>
+struct negation
+ // NOLINTNEXTLINE
+ : std::integral_constant<bool, bool(!P::value)> {};
+
+// Base case: with zero predicates the answer is always true.
+template <typename...>
+struct conjunction : std::true_type {};
+
+// With a single predicate, the answer is that predicate.
+template <typename P1>
+struct conjunction<P1> : P1 {};
+
+// With multiple predicates the answer is the first predicate if that is false,
+// and we recurse otherwise.
+template <typename P1, typename... Ps>
+struct conjunction<P1, Ps...>
+ : std::conditional<bool(P1::value), conjunction<Ps...>, P1>::type {};
+
+template <typename...>
+struct disjunction : std::false_type {};
+
+template <typename P1>
+struct disjunction<P1> : P1 {};
+
+template <typename P1, typename... Ps>
+struct disjunction<P1, Ps...>
+ // NOLINTNEXTLINE
+ : std::conditional<!bool(P1::value), disjunction<Ps...>, P1>::type {};
+
+template <typename...>
+using void_t = void;
+
+// Detects whether an expression of type `From` can be implicitly converted to
+// `To` according to [conv]. In C++17, [conv]/3 defines this as follows:
+//
+// An expression e can be implicitly converted to a type T if and only if
+// the declaration T t=e; is well-formed, for some invented temporary
+// variable t ([dcl.init]).
+//
+// [conv]/2 implies we can use function argument passing to detect whether this
+// initialization is valid.
+//
+// Note that this is distinct from is_convertible, which requires this be valid:
+//
+// To test() {
+// return declval<From>();
+// }
+//
+// In particular, is_convertible doesn't give the correct answer when `To` and
+// `From` are the same non-moveable type since `declval<From>` will be an rvalue
+// reference, defeating the guaranteed copy elision that would otherwise make
+// this function work.
+//
+// REQUIRES: `From` is not cv void.
+template <typename From, typename To>
+struct is_implicitly_convertible {
+ private:
+ // A function that accepts a parameter of type T. This can be called with type
+ // U successfully only if U is implicitly convertible to T.
+ template <typename T>
+ static void Accept(T);
+
+ // A function that creates a value of type T.
+ template <typename T>
+ static T Make();
+
+ // An overload be selected when implicit conversion from T to To is possible.
+ template <typename T, typename = decltype(Accept<To>(Make<T>()))>
+ static std::true_type TestImplicitConversion(int);
+
+ // A fallback overload selected in all other cases.
+ template <typename T>
+ static std::false_type TestImplicitConversion(...);
+
+ public:
+ using type = decltype(TestImplicitConversion<From>(0));
+ static constexpr bool value = type::value;
+};
+
+// Like std::invoke_result_t from C++17, but works only for objects with call
+// operators (not e.g. member function pointers, which we don't need specific
+// support for in OnceAction because std::function deals with them).
+template <typename F, typename... Args>
+using call_result_t = decltype(std::declval<F>()(std::declval<Args>()...));
+
+template <typename Void, typename R, typename F, typename... Args>
+struct is_callable_r_impl : std::false_type {};
+
+// Specialize the struct for those template arguments where call_result_t is
+// well-formed. When it's not, the generic template above is chosen, resulting
+// in std::false_type.
+template <typename R, typename F, typename... Args>
+struct is_callable_r_impl<void_t<call_result_t<F, Args...>>, R, F, Args...>
+ : std::conditional<
+ std::is_void<R>::value, //
+ std::true_type, //
+ is_implicitly_convertible<call_result_t<F, Args...>, R>>::type {};
+
+// Like std::is_invocable_r from C++17, but works only for objects with call
+// operators. See the note on call_result_t.
+template <typename R, typename F, typename... Args>
+using is_callable_r = is_callable_r_impl<void, R, F, Args...>;
+
+// Like std::as_const from C++17.
+template <typename T>
+typename std::add_const<T>::type& as_const(T& t) {
+ return t;
+}
} // namespace internal
+// Specialized for function types below.
+template <typename F>
+class OnceAction;
+
+// An action that can only be used once.
+//
+// This is accepted by WillOnce, which doesn't require the underlying action to
+// be copy-constructible (only move-constructible), and promises to invoke it as
+// an rvalue reference. This allows the action to work with move-only types like
+// std::move_only_function in a type-safe manner.
+//
+// For example:
+//
+// // Assume we have some API that needs to accept a unique pointer to some
+// // non-copyable object Foo.
+// void AcceptUniquePointer(std::unique_ptr<Foo> foo);
+//
+// // We can define an action that provides a Foo to that API. Because It
+// // has to give away its unique pointer, it must not be called more than
+// // once, so its call operator is &&-qualified.
+// struct ProvideFoo {
+// std::unique_ptr<Foo> foo;
+//
+// void operator()() && {
+// AcceptUniquePointer(std::move(Foo));
+// }
+// };
+//
+// // This action can be used with WillOnce.
+// EXPECT_CALL(mock, Call)
+// .WillOnce(ProvideFoo{std::make_unique<Foo>(...)});
+//
+// // But a call to WillRepeatedly will fail to compile. This is correct,
+// // since the action cannot correctly be used repeatedly.
+// EXPECT_CALL(mock, Call)
+// .WillRepeatedly(ProvideFoo{std::make_unique<Foo>(...)});
+//
+// A less-contrived example would be an action that returns an arbitrary type,
+// whose &&-qualified call operator is capable of dealing with move-only types.
+template <typename Result, typename... Args>
+class OnceAction<Result(Args...)> final {
+ private:
+ // True iff we can use the given callable type (or lvalue reference) directly
+ // via StdFunctionAdaptor.
+ template <typename Callable>
+ using IsDirectlyCompatible = internal::conjunction<
+ // It must be possible to capture the callable in StdFunctionAdaptor.
+ std::is_constructible<typename std::decay<Callable>::type, Callable>,
+ // The callable must be compatible with our signature.
+ internal::is_callable_r<Result, typename std::decay<Callable>::type,
+ Args...>>;
+
+ // True iff we can use the given callable type via StdFunctionAdaptor once we
+ // ignore incoming arguments.
+ template <typename Callable>
+ using IsCompatibleAfterIgnoringArguments = internal::conjunction<
+ // It must be possible to capture the callable in a lambda.
+ std::is_constructible<typename std::decay<Callable>::type, Callable>,
+ // The callable must be invocable with zero arguments, returning something
+ // convertible to Result.
+ internal::is_callable_r<Result, typename std::decay<Callable>::type>>;
+
+ public:
+ // Construct from a callable that is directly compatible with our mocked
+ // signature: it accepts our function type's arguments and returns something
+ // convertible to our result type.
+ template <typename Callable,
+ typename std::enable_if<
+ internal::conjunction<
+ // Teach clang on macOS that we're not talking about a
+ // copy/move constructor here. Otherwise it gets confused
+ // when checking the is_constructible requirement of our
+ // traits above.
+ internal::negation<std::is_same<
+ OnceAction, typename std::decay<Callable>::type>>,
+ IsDirectlyCompatible<Callable>> //
+ ::value,
+ int>::type = 0>
+ OnceAction(Callable&& callable) // NOLINT
+ : function_(StdFunctionAdaptor<typename std::decay<Callable>::type>(
+ {}, std::forward<Callable>(callable))) {}
+
+ // As above, but for a callable that ignores the mocked function's arguments.
+ template <typename Callable,
+ typename std::enable_if<
+ internal::conjunction<
+ // Teach clang on macOS that we're not talking about a
+ // copy/move constructor here. Otherwise it gets confused
+ // when checking the is_constructible requirement of our
+ // traits above.
+ internal::negation<std::is_same<
+ OnceAction, typename std::decay<Callable>::type>>,
+ // Exclude callables for which the overload above works.
+ // We'd rather provide the arguments if possible.
+ internal::negation<IsDirectlyCompatible<Callable>>,
+ IsCompatibleAfterIgnoringArguments<Callable>>::value,
+ int>::type = 0>
+ OnceAction(Callable&& callable) // NOLINT
+ // Call the constructor above with a callable
+ // that ignores the input arguments.
+ : OnceAction(IgnoreIncomingArguments<typename std::decay<Callable>::type>{
+ std::forward<Callable>(callable)}) {}
+
+ // We are naturally copyable because we store only an std::function, but
+ // semantically we should not be copyable.
+ OnceAction(const OnceAction&) = delete;
+ OnceAction& operator=(const OnceAction&) = delete;
+ OnceAction(OnceAction&&) = default;
+
+ // Invoke the underlying action callable with which we were constructed,
+ // handing it the supplied arguments.
+ Result Call(Args... args) && {
+ return function_(std::forward<Args>(args)...);
+ }
+
+ private:
+ // An adaptor that wraps a callable that is compatible with our signature and
+ // being invoked as an rvalue reference so that it can be used as an
+ // StdFunctionAdaptor. This throws away type safety, but that's fine because
+ // this is only used by WillOnce, which we know calls at most once.
+ //
+ // Once we have something like std::move_only_function from C++23, we can do
+ // away with this.
+ template <typename Callable>
+ class StdFunctionAdaptor final {
+ public:
+ // A tag indicating that the (otherwise universal) constructor is accepting
+ // the callable itself, instead of e.g. stealing calls for the move
+ // constructor.
+ struct CallableTag final {};
+
+ template <typename F>
+ explicit StdFunctionAdaptor(CallableTag, F&& callable)
+ : callable_(std::make_shared<Callable>(std::forward<F>(callable))) {}
+
+ // Rather than explicitly returning Result, we return whatever the wrapped
+ // callable returns. This allows for compatibility with existing uses like
+ // the following, when the mocked function returns void:
+ //
+ // EXPECT_CALL(mock_fn_, Call)
+ // .WillOnce([&] {
+ // [...]
+ // return 0;
+ // });
+ //
+ // Such a callable can be turned into std::function<void()>. If we use an
+ // explicit return type of Result here then it *doesn't* work with
+ // std::function, because we'll get a "void function should not return a
+ // value" error.
+ //
+ // We need not worry about incompatible result types because the SFINAE on
+ // OnceAction already checks this for us. std::is_invocable_r_v itself makes
+ // the same allowance for void result types.
+ template <typename... ArgRefs>
+ internal::call_result_t<Callable, ArgRefs...> operator()(
+ ArgRefs&&... args) const {
+ return std::move(*callable_)(std::forward<ArgRefs>(args)...);
+ }
+
+ private:
+ // We must put the callable on the heap so that we are copyable, which
+ // std::function needs.
+ std::shared_ptr<Callable> callable_;
+ };
+
+ // An adaptor that makes a callable that accepts zero arguments callable with
+ // our mocked arguments.
+ template <typename Callable>
+ struct IgnoreIncomingArguments {
+ internal::call_result_t<Callable> operator()(Args&&...) {
+ return std::move(callable)();
+ }
+
+ Callable callable;
+ };
+
+ std::function<Result(Args...)> function_;
+};
+
// When an unexpected function call is encountered, Google Mock will
// let it return a default value if the user has specified one for its
// return type, or if the return type has a built-in default value;
@@ -339,7 +625,8 @@
private:
const T value_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(FixedValueProducer);
+ FixedValueProducer(const FixedValueProducer&) = delete;
+ FixedValueProducer& operator=(const FixedValueProducer&) = delete;
};
class FactoryValueProducer : public ValueProducer {
@@ -350,7 +637,8 @@
private:
const FactoryFunction factory_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(FactoryValueProducer);
+ FactoryValueProducer(const FactoryValueProducer&) = delete;
+ FactoryValueProducer& operator=(const FactoryValueProducer&) = delete;
};
static ValueProducer* producer_;
@@ -424,28 +712,34 @@
virtual Result Perform(const ArgumentTuple& args) = 0;
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ActionInterface);
+ ActionInterface(const ActionInterface&) = delete;
+ ActionInterface& operator=(const ActionInterface&) = delete;
};
-// An Action<F> is a copyable and IMMUTABLE (except by assignment)
-// object that represents an action to be taken when a mock function
-// of type F is called. The implementation of Action<T> is just a
-// std::shared_ptr to const ActionInterface<T>. Don't inherit from Action!
-// You can view an object implementing ActionInterface<F> as a
-// concrete action (including its current state), and an Action<F>
-// object as a handle to it.
template <typename F>
-class Action {
+class Action;
+
+// An Action<R(Args...)> is a copyable and IMMUTABLE (except by assignment)
+// object that represents an action to be taken when a mock function of type
+// R(Args...) is called. The implementation of Action<T> is just a
+// std::shared_ptr to const ActionInterface<T>. Don't inherit from Action! You
+// can view an object implementing ActionInterface<F> as a concrete action
+// (including its current state), and an Action<F> object as a handle to it.
+template <typename R, typename... Args>
+class Action<R(Args...)> {
+ private:
+ using F = R(Args...);
+
// Adapter class to allow constructing Action from a legacy ActionInterface.
// New code should create Actions from functors instead.
struct ActionAdapter {
// Adapter must be copyable to satisfy std::function requirements.
::std::shared_ptr<ActionInterface<F>> impl_;
- template <typename... Args>
- typename internal::Function<F>::Result operator()(Args&&... args) {
+ template <typename... InArgs>
+ typename internal::Function<F>::Result operator()(InArgs&&... args) {
return impl_->Perform(
- ::std::forward_as_tuple(::std::forward<Args>(args)...));
+ ::std::forward_as_tuple(::std::forward<InArgs>(args)...));
}
};
@@ -480,7 +774,8 @@
// Action<F>, as long as F's arguments can be implicitly converted
// to Func's and Func's return type can be implicitly converted to F's.
template <typename Func>
- explicit Action(const Action<Func>& action) : fun_(action.fun_) {}
+ Action(const Action<Func>& action) // NOLINT
+ : fun_(action.fun_) {}
// Returns true if and only if this is the DoDefault() action.
bool IsDoDefault() const { return fun_ == nullptr; }
@@ -498,6 +793,24 @@
return internal::Apply(fun_, ::std::move(args));
}
+ // An action can be used as a OnceAction, since it's obviously safe to call it
+ // once.
+ operator OnceAction<F>() const { // NOLINT
+ // Return a OnceAction-compatible callable that calls Perform with the
+ // arguments it is provided. We could instead just return fun_, but then
+ // we'd need to handle the IsDoDefault() case separately.
+ struct OA {
+ Action<F> action;
+
+ R operator()(Args... args) && {
+ return action.Perform(
+ std::forward_as_tuple(std::forward<Args>(args)...));
+ }
+ };
+
+ return OA{*this};
+ }
+
private:
template <typename G>
friend class Action;
@@ -514,8 +827,8 @@
template <typename FunctionImpl>
struct IgnoreArgs {
- template <typename... Args>
- Result operator()(const Args&...) const {
+ template <typename... InArgs>
+ Result operator()(const InArgs&...) const {
return function_impl();
}
@@ -606,118 +919,198 @@
T payload;
};
-// Implements the polymorphic Return(x) action, which can be used in
-// any function that returns the type of x, regardless of the argument
-// types.
-//
-// Note: The value passed into Return must be converted into
-// Function<F>::Result when this action is cast to Action<F> rather than
-// when that action is performed. This is important in scenarios like
-//
-// MOCK_METHOD1(Method, T(U));
-// ...
-// {
-// Foo foo;
-// X x(&foo);
-// EXPECT_CALL(mock, Method(_)).WillOnce(Return(x));
-// }
-//
-// In the example above the variable x holds reference to foo which leaves
-// scope and gets destroyed. If copying X just copies a reference to foo,
-// that copy will be left with a hanging reference. If conversion to T
-// makes a copy of foo, the above code is safe. To support that scenario, we
-// need to make sure that the type conversion happens inside the EXPECT_CALL
-// statement, and conversion of the result of Return to Action<T(U)> is a
-// good place for that.
-//
-// The real life example of the above scenario happens when an invocation
-// of gtl::Container() is passed into Return.
-//
+// The general implementation of Return(R). Specializations follow below.
template <typename R>
-class ReturnAction {
+class ReturnAction final {
public:
- // Constructs a ReturnAction object from the value to be returned.
- // 'value' is passed by value instead of by const reference in order
- // to allow Return("string literal") to compile.
- explicit ReturnAction(R value) : value_(new R(std::move(value))) {}
+ explicit ReturnAction(R value) : value_(std::move(value)) {}
- // This template type conversion operator allows Return(x) to be
- // used in ANY function that returns x's type.
- template <typename F>
- operator Action<F>() const { // NOLINT
- // Assert statement belongs here because this is the best place to verify
- // conditions on F. It produces the clearest error messages
- // in most compilers.
- // Impl really belongs in this scope as a local class but can't
- // because MSVC produces duplicate symbols in different translation units
- // in this case. Until MS fixes that bug we put Impl into the class scope
- // and put the typedef both here (for use in assert statement) and
- // in the Impl class. But both definitions must be the same.
- typedef typename Function<F>::Result Result;
- GTEST_COMPILE_ASSERT_(
- !std::is_reference<Result>::value,
- use_ReturnRef_instead_of_Return_to_return_a_reference);
- static_assert(!std::is_void<Result>::value,
- "Can't use Return() on an action expected to return `void`.");
- return Action<F>(new Impl<R, F>(value_));
+ template <typename U, typename... Args,
+ typename = typename std::enable_if<conjunction<
+ // See the requirements documented on Return.
+ negation<std::is_same<void, U>>, //
+ negation<std::is_reference<U>>, //
+ std::is_convertible<R, U>, //
+ std::is_move_constructible<U>>::value>::type>
+ operator OnceAction<U(Args...)>() && { // NOLINT
+ return Impl<U>(std::move(value_));
+ }
+
+ template <typename U, typename... Args,
+ typename = typename std::enable_if<conjunction<
+ // See the requirements documented on Return.
+ negation<std::is_same<void, U>>, //
+ negation<std::is_reference<U>>, //
+ std::is_convertible<const R&, U>, //
+ std::is_copy_constructible<U>>::value>::type>
+ operator Action<U(Args...)>() const { // NOLINT
+ return Impl<U>(value_);
}
private:
- // Implements the Return(x) action for a particular function type F.
- template <typename R_, typename F>
- class Impl : public ActionInterface<F> {
+ // Implements the Return(x) action for a mock function that returns type U.
+ template <typename U>
+ class Impl final {
public:
- typedef typename Function<F>::Result Result;
- typedef typename Function<F>::ArgumentTuple ArgumentTuple;
+ // The constructor used when the return value is allowed to move from the
+ // input value (i.e. we are converting to OnceAction).
+ explicit Impl(R&& input_value)
+ : state_(new State(std::move(input_value))) {}
- // The implicit cast is necessary when Result has more than one
- // single-argument constructor (e.g. Result is std::vector<int>) and R
- // has a type conversion operator template. In that case, value_(value)
- // won't compile as the compiler doesn't known which constructor of
- // Result to call. ImplicitCast_ forces the compiler to convert R to
- // Result without considering explicit constructors, thus resolving the
- // ambiguity. value_ is then initialized using its copy constructor.
- explicit Impl(const std::shared_ptr<R>& value)
- : value_before_cast_(*value),
- value_(ImplicitCast_<Result>(value_before_cast_)) {}
+ // The constructor used when the return value is not allowed to move from
+ // the input value (i.e. we are converting to Action).
+ explicit Impl(const R& input_value) : state_(new State(input_value)) {}
- Result Perform(const ArgumentTuple&) override { return value_; }
+ U operator()() && { return std::move(state_->value); }
+ U operator()() const& { return state_->value; }
private:
- GTEST_COMPILE_ASSERT_(!std::is_reference<Result>::value,
- Result_cannot_be_a_reference_type);
- // We save the value before casting just in case it is being cast to a
- // wrapper type.
- R value_before_cast_;
- Result value_;
+ // We put our state on the heap so that the compiler-generated copy/move
+ // constructors work correctly even when U is a reference-like type. This is
+ // necessary only because we eagerly create State::value (see the note on
+ // that symbol for details). If we instead had only the input value as a
+ // member then the default constructors would work fine.
+ //
+ // For example, when R is std::string and U is std::string_view, value is a
+ // reference to the string backed by input_value. The copy constructor would
+ // copy both, so that we wind up with a new input_value object (with the
+ // same contents) and a reference to the *old* input_value object rather
+ // than the new one.
+ struct State {
+ explicit State(const R& input_value_in)
+ : input_value(input_value_in),
+ // Make an implicit conversion to Result before initializing the U
+ // object we store, avoiding calling any explicit constructor of U
+ // from R.
+ //
+ // This simulates the language rules: a function with return type U
+ // that does `return R()` requires R to be implicitly convertible to
+ // U, and uses that path for the conversion, even U Result has an
+ // explicit constructor from R.
+ value(ImplicitCast_<U>(internal::as_const(input_value))) {}
- GTEST_DISALLOW_COPY_AND_ASSIGN_(Impl);
+ // As above, but for the case where we're moving from the ReturnAction
+ // object because it's being used as a OnceAction.
+ explicit State(R&& input_value_in)
+ : input_value(std::move(input_value_in)),
+ // For the same reason as above we make an implicit conversion to U
+ // before initializing the value.
+ //
+ // Unlike above we provide the input value as an rvalue to the
+ // implicit conversion because this is a OnceAction: it's fine if it
+ // wants to consume the input value.
+ value(ImplicitCast_<U>(std::move(input_value))) {}
+
+ // A copy of the value originally provided by the user. We retain this in
+ // addition to the value of the mock function's result type below in case
+ // the latter is a reference-like type. See the std::string_view example
+ // in the documentation on Return.
+ R input_value;
+
+ // The value we actually return, as the type returned by the mock function
+ // itself.
+ //
+ // We eagerly initialize this here, rather than lazily doing the implicit
+ // conversion automatically each time Perform is called, for historical
+ // reasons: in 2009-11, commit a070cbd91c (Google changelist 13540126)
+ // made the Action<U()> conversion operator eagerly convert the R value to
+ // U, but without keeping the R alive. This broke the use case discussed
+ // in the documentation for Return, making reference-like types such as
+ // std::string_view not safe to use as U where the input type R is a
+ // value-like type such as std::string.
+ //
+ // The example the commit gave was not very clear, nor was the issue
+ // thread (https://github.com/google/googlemock/issues/86), but it seems
+ // the worry was about reference-like input types R that flatten to a
+ // value-like type U when being implicitly converted. An example of this
+ // is std::vector<bool>::reference, which is often a proxy type with an
+ // reference to the underlying vector:
+ //
+ // // Helper method: have the mock function return bools according
+ // // to the supplied script.
+ // void SetActions(MockFunction<bool(size_t)>& mock,
+ // const std::vector<bool>& script) {
+ // for (size_t i = 0; i < script.size(); ++i) {
+ // EXPECT_CALL(mock, Call(i)).WillOnce(Return(script[i]));
+ // }
+ // }
+ //
+ // TEST(Foo, Bar) {
+ // // Set actions using a temporary vector, whose operator[]
+ // // returns proxy objects that references that will be
+ // // dangling once the call to SetActions finishes and the
+ // // vector is destroyed.
+ // MockFunction<bool(size_t)> mock;
+ // SetActions(mock, {false, true});
+ //
+ // EXPECT_FALSE(mock.AsStdFunction()(0));
+ // EXPECT_TRUE(mock.AsStdFunction()(1));
+ // }
+ //
+ // This eager conversion helps with a simple case like this, but doesn't
+ // fully make these types work in general. For example the following still
+ // uses a dangling reference:
+ //
+ // TEST(Foo, Baz) {
+ // MockFunction<std::vector<std::string>()> mock;
+ //
+ // // Return the same vector twice, and then the empty vector
+ // // thereafter.
+ // auto action = Return(std::initializer_list<std::string>{
+ // "taco", "burrito",
+ // });
+ //
+ // EXPECT_CALL(mock, Call)
+ // .WillOnce(action)
+ // .WillOnce(action)
+ // .WillRepeatedly(Return(std::vector<std::string>{}));
+ //
+ // EXPECT_THAT(mock.AsStdFunction()(),
+ // ElementsAre("taco", "burrito"));
+ // EXPECT_THAT(mock.AsStdFunction()(),
+ // ElementsAre("taco", "burrito"));
+ // EXPECT_THAT(mock.AsStdFunction()(), IsEmpty());
+ // }
+ //
+ U value;
+ };
+
+ const std::shared_ptr<State> state_;
};
- // Partially specialize for ByMoveWrapper. This version of ReturnAction will
- // move its contents instead.
- template <typename R_, typename F>
- class Impl<ByMoveWrapper<R_>, F> : public ActionInterface<F> {
- public:
- typedef typename Function<F>::Result Result;
- typedef typename Function<F>::ArgumentTuple ArgumentTuple;
+ R value_;
+};
- explicit Impl(const std::shared_ptr<R>& wrapper)
- : performed_(false), wrapper_(wrapper) {}
+// A specialization of ReturnAction<R> when R is ByMoveWrapper<T> for some T.
+//
+// This version applies the type system-defeating hack of moving from T even in
+// the const call operator, checking at runtime that it isn't called more than
+// once, since the user has declared their intent to do so by using ByMove.
+template <typename T>
+class ReturnAction<ByMoveWrapper<T>> final {
+ public:
+ explicit ReturnAction(ByMoveWrapper<T> wrapper)
+ : state_(new State(std::move(wrapper.payload))) {}
- Result Perform(const ArgumentTuple&) override {
- GTEST_CHECK_(!performed_)
- << "A ByMove() action should only be performed once.";
- performed_ = true;
- return std::move(wrapper_->payload);
- }
+ T operator()() const {
+ GTEST_CHECK_(!state_->called)
+ << "A ByMove() action must be performed at most once.";
- private:
- bool performed_;
- const std::shared_ptr<R> wrapper_;
+ state_->called = true;
+ return std::move(state_->value);
+ }
+
+ private:
+ // We store our state on the heap so that we are copyable as required by
+ // Action, despite the fact that we are stateful and T may not be copyable.
+ struct State {
+ explicit State(T&& value_in) : value(std::move(value_in)) {}
+
+ T value;
+ bool called = false;
};
- const std::shared_ptr<R> value_;
+ const std::shared_ptr<State> state_;
};
// Implements the ReturnNull() action.
@@ -759,8 +1152,8 @@
// Asserts that the function return type is a reference. This
// catches the user error of using ReturnRef(x) when Return(x)
// should be used, and generates some helpful error message.
- GTEST_COMPILE_ASSERT_(std::is_reference<Result>::value,
- use_Return_instead_of_ReturnRef_to_return_a_value);
+ static_assert(std::is_reference<Result>::value,
+ "use Return instead of ReturnRef to return a value");
return Action<F>(new Impl<F>(ref_));
}
@@ -801,9 +1194,8 @@
// Asserts that the function return type is a reference. This
// catches the user error of using ReturnRefOfCopy(x) when Return(x)
// should be used, and generates some helpful error message.
- GTEST_COMPILE_ASSERT_(
- std::is_reference<Result>::value,
- use_Return_instead_of_ReturnRefOfCopy_to_return_a_value);
+ static_assert(std::is_reference<Result>::value,
+ "use Return instead of ReturnRefOfCopy to return a value");
return Action<F>(new Impl<F>(value_));
}
@@ -839,7 +1231,7 @@
template <typename... Args>
T operator()(Args&&...) const {
- return state_->Next();
+ return state_->Next();
}
private:
@@ -862,7 +1254,9 @@
// This template type conversion operator allows DoDefault() to be
// used in any function.
template <typename F>
- operator Action<F>() const { return Action<F>(); } // NOLINT
+ operator Action<F>() const {
+ return Action<F>();
+ } // NOLINT
};
// Implements the Assign action to set a given pointer referent to a
@@ -890,8 +1284,7 @@
class SetErrnoAndReturnAction {
public:
SetErrnoAndReturnAction(int errno_value, T result)
- : errno_(errno_value),
- result_(result) {}
+ : errno_(errno_value), result_(result) {}
template <typename Result, typename ArgumentTuple>
Result Perform(const ArgumentTuple& /* args */) const {
errno = errno_;
@@ -1002,8 +1395,8 @@
private:
// Type OriginalFunction is the same as F except that its return
// type is IgnoredValue.
- typedef typename internal::Function<F>::MakeResultIgnoredValue
- OriginalFunction;
+ typedef
+ typename internal::Function<F>::MakeResultIgnoredValue OriginalFunction;
const Action<OriginalFunction> action_;
};
@@ -1013,55 +1406,239 @@
template <typename InnerAction, size_t... I>
struct WithArgsAction {
- InnerAction action;
+ InnerAction inner_action;
- // The inner action could be anything convertible to Action<X>.
- // We use the conversion operator to detect the signature of the inner Action.
+ // The signature of the function as seen by the inner action, given an out
+ // action with the given result and argument types.
template <typename R, typename... Args>
- operator Action<R(Args...)>() const { // NOLINT
- using TupleType = std::tuple<Args...>;
- Action<R(typename std::tuple_element<I, TupleType>::type...)>
- converted(action);
+ using InnerSignature =
+ R(typename std::tuple_element<I, std::tuple<Args...>>::type...);
- return [converted](Args... args) -> R {
+ // Rather than a call operator, we must define conversion operators to
+ // particular action types. This is necessary for embedded actions like
+ // DoDefault(), which rely on an action conversion operators rather than
+ // providing a call operator because even with a particular set of arguments
+ // they don't have a fixed return type.
+
+ template <typename R, typename... Args,
+ typename std::enable_if<
+ std::is_convertible<
+ InnerAction,
+ // Unfortunately we can't use the InnerSignature alias here;
+ // MSVC complains about the I parameter pack not being
+ // expanded (error C3520) despite it being expanded in the
+ // type alias.
+ OnceAction<R(typename std::tuple_element<
+ I, std::tuple<Args...>>::type...)>>::value,
+ int>::type = 0>
+ operator OnceAction<R(Args...)>() && { // NOLINT
+ struct OA {
+ OnceAction<InnerSignature<R, Args...>> inner_action;
+
+ R operator()(Args&&... args) && {
+ return std::move(inner_action)
+ .Call(std::get<I>(
+ std::forward_as_tuple(std::forward<Args>(args)...))...);
+ }
+ };
+
+ return OA{std::move(inner_action)};
+ }
+
+ template <typename R, typename... Args,
+ typename std::enable_if<
+ std::is_convertible<
+ const InnerAction&,
+ // Unfortunately we can't use the InnerSignature alias here;
+ // MSVC complains about the I parameter pack not being
+ // expanded (error C3520) despite it being expanded in the
+ // type alias.
+ Action<R(typename std::tuple_element<
+ I, std::tuple<Args...>>::type...)>>::value,
+ int>::type = 0>
+ operator Action<R(Args...)>() const { // NOLINT
+ Action<InnerSignature<R, Args...>> converted(inner_action);
+
+ return [converted](Args&&... args) -> R {
return converted.Perform(std::forward_as_tuple(
- std::get<I>(std::forward_as_tuple(std::forward<Args>(args)...))...));
+ std::get<I>(std::forward_as_tuple(std::forward<Args>(args)...))...));
};
}
};
template <typename... Actions>
-struct DoAllAction {
- private:
+class DoAllAction;
+
+// Base case: only a single action.
+template <typename FinalAction>
+class DoAllAction<FinalAction> {
+ public:
+ struct UserConstructorTag {};
+
template <typename T>
- using NonFinalType =
+ explicit DoAllAction(UserConstructorTag, T&& action)
+ : final_action_(std::forward<T>(action)) {}
+
+ // Rather than a call operator, we must define conversion operators to
+ // particular action types. This is necessary for embedded actions like
+ // DoDefault(), which rely on an action conversion operators rather than
+ // providing a call operator because even with a particular set of arguments
+ // they don't have a fixed return type.
+
+ template <typename R, typename... Args,
+ typename std::enable_if<
+ std::is_convertible<FinalAction, OnceAction<R(Args...)>>::value,
+ int>::type = 0>
+ operator OnceAction<R(Args...)>() && { // NOLINT
+ return std::move(final_action_);
+ }
+
+ template <
+ typename R, typename... Args,
+ typename std::enable_if<
+ std::is_convertible<const FinalAction&, Action<R(Args...)>>::value,
+ int>::type = 0>
+ operator Action<R(Args...)>() const { // NOLINT
+ return final_action_;
+ }
+
+ private:
+ FinalAction final_action_;
+};
+
+// Recursive case: support N actions by calling the initial action and then
+// calling through to the base class containing N-1 actions.
+template <typename InitialAction, typename... OtherActions>
+class DoAllAction<InitialAction, OtherActions...>
+ : private DoAllAction<OtherActions...> {
+ private:
+ using Base = DoAllAction<OtherActions...>;
+
+ // The type of reference that should be provided to an initial action for a
+ // mocked function parameter of type T.
+ //
+ // There are two quirks here:
+ //
+ // * Unlike most forwarding functions, we pass scalars through by value.
+ // This isn't strictly necessary because an lvalue reference would work
+ // fine too and be consistent with other non-reference types, but it's
+ // perhaps less surprising.
+ //
+ // For example if the mocked function has signature void(int), then it
+ // might seem surprising for the user's initial action to need to be
+ // convertible to Action<void(const int&)>. This is perhaps less
+ // surprising for a non-scalar type where there may be a performance
+ // impact, or it might even be impossible, to pass by value.
+ //
+ // * More surprisingly, `const T&` is often not a const reference type.
+ // By the reference collapsing rules in C++17 [dcl.ref]/6, if T refers to
+ // U& or U&& for some non-scalar type U, then InitialActionArgType<T> is
+ // U&. In other words, we may hand over a non-const reference.
+ //
+ // So for example, given some non-scalar type Obj we have the following
+ // mappings:
+ //
+ // T InitialActionArgType<T>
+ // ------- -----------------------
+ // Obj const Obj&
+ // Obj& Obj&
+ // Obj&& Obj&
+ // const Obj const Obj&
+ // const Obj& const Obj&
+ // const Obj&& const Obj&
+ //
+ // In other words, the initial actions get a mutable view of an non-scalar
+ // argument if and only if the mock function itself accepts a non-const
+ // reference type. They are never given an rvalue reference to an
+ // non-scalar type.
+ //
+ // This situation makes sense if you imagine use with a matcher that is
+ // designed to write through a reference. For example, if the caller wants
+ // to fill in a reference argument and then return a canned value:
+ //
+ // EXPECT_CALL(mock, Call)
+ // .WillOnce(DoAll(SetArgReferee<0>(17), Return(19)));
+ //
+ template <typename T>
+ using InitialActionArgType =
typename std::conditional<std::is_scalar<T>::value, T, const T&>::type;
- template <typename ActionT, size_t... I>
- std::vector<ActionT> Convert(IndexSequence<I...>) const {
- return {ActionT(std::get<I>(actions))...};
- }
-
public:
- std::tuple<Actions...> actions;
+ struct UserConstructorTag {};
- template <typename R, typename... Args>
- operator Action<R(Args...)>() const { // NOLINT
- struct Op {
- std::vector<Action<void(NonFinalType<Args>...)>> converted;
- Action<R(Args...)> last;
- R operator()(Args... args) const {
- auto tuple_args = std::forward_as_tuple(std::forward<Args>(args)...);
- for (auto& a : converted) {
- a.Perform(tuple_args);
- }
- return last.Perform(std::move(tuple_args));
+ template <typename T, typename... U>
+ explicit DoAllAction(UserConstructorTag, T&& initial_action,
+ U&&... other_actions)
+ : Base({}, std::forward<U>(other_actions)...),
+ initial_action_(std::forward<T>(initial_action)) {}
+
+ template <typename R, typename... Args,
+ typename std::enable_if<
+ conjunction<
+ // Both the initial action and the rest must support
+ // conversion to OnceAction.
+ std::is_convertible<
+ InitialAction,
+ OnceAction<void(InitialActionArgType<Args>...)>>,
+ std::is_convertible<Base, OnceAction<R(Args...)>>>::value,
+ int>::type = 0>
+ operator OnceAction<R(Args...)>() && { // NOLINT
+ // Return an action that first calls the initial action with arguments
+ // filtered through InitialActionArgType, then forwards arguments directly
+ // to the base class to deal with the remaining actions.
+ struct OA {
+ OnceAction<void(InitialActionArgType<Args>...)> initial_action;
+ OnceAction<R(Args...)> remaining_actions;
+
+ R operator()(Args... args) && {
+ std::move(initial_action)
+ .Call(static_cast<InitialActionArgType<Args>>(args)...);
+
+ return std::move(remaining_actions).Call(std::forward<Args>(args)...);
}
};
- return Op{Convert<Action<void(NonFinalType<Args>...)>>(
- MakeIndexSequence<sizeof...(Actions) - 1>()),
- std::get<sizeof...(Actions) - 1>(actions)};
+
+ return OA{
+ std::move(initial_action_),
+ std::move(static_cast<Base&>(*this)),
+ };
}
+
+ template <
+ typename R, typename... Args,
+ typename std::enable_if<
+ conjunction<
+ // Both the initial action and the rest must support conversion to
+ // Action.
+ std::is_convertible<const InitialAction&,
+ Action<void(InitialActionArgType<Args>...)>>,
+ std::is_convertible<const Base&, Action<R(Args...)>>>::value,
+ int>::type = 0>
+ operator Action<R(Args...)>() const { // NOLINT
+ // Return an action that first calls the initial action with arguments
+ // filtered through InitialActionArgType, then forwards arguments directly
+ // to the base class to deal with the remaining actions.
+ struct OA {
+ Action<void(InitialActionArgType<Args>...)> initial_action;
+ Action<R(Args...)> remaining_actions;
+
+ R operator()(Args... args) const {
+ initial_action.Perform(std::forward_as_tuple(
+ static_cast<InitialActionArgType<Args>>(args)...));
+
+ return remaining_actions.Perform(
+ std::forward_as_tuple(std::forward<Args>(args)...));
+ }
+ };
+
+ return OA{
+ initial_action_,
+ static_cast<const Base&>(*this),
+ };
+ }
+
+ private:
+ InitialAction initial_action_;
};
template <typename T, typename... Params>
@@ -1078,10 +1655,11 @@
template <size_t k>
struct ReturnArgAction {
- template <typename... Args>
- auto operator()(const Args&... args) const ->
- typename std::tuple_element<k, std::tuple<Args...>>::type {
- return std::get<k>(std::tie(args...));
+ template <typename... Args,
+ typename = typename std::enable_if<(k < sizeof...(Args))>::type>
+ auto operator()(Args&&... args) const -> decltype(std::get<k>(
+ std::forward_as_tuple(std::forward<Args>(args)...))) {
+ return std::get<k>(std::forward_as_tuple(std::forward<Args>(args)...));
}
};
@@ -1203,7 +1781,8 @@
template <typename... Action>
internal::DoAllAction<typename std::decay<Action>::type...> DoAll(
Action&&... action) {
- return {std::forward_as_tuple(std::forward<Action>(action)...)};
+ return internal::DoAllAction<typename std::decay<Action>::type...>(
+ {}, std::forward<Action>(action)...);
}
// WithArg<k>(an_action) creates an action that passes the k-th
@@ -1212,8 +1791,8 @@
// multiple arguments. For convenience, we also provide
// WithArgs<k>(an_action) (defined below) as a synonym.
template <size_t k, typename InnerAction>
-internal::WithArgsAction<typename std::decay<InnerAction>::type, k>
-WithArg(InnerAction&& action) {
+internal::WithArgsAction<typename std::decay<InnerAction>::type, k> WithArg(
+ InnerAction&& action) {
return {std::forward<InnerAction>(action)};
}
@@ -1232,14 +1811,35 @@
// argument. In other words, it adapts an action accepting no
// argument to one that accepts (and ignores) arguments.
template <typename InnerAction>
-internal::WithArgsAction<typename std::decay<InnerAction>::type>
-WithoutArgs(InnerAction&& action) {
+internal::WithArgsAction<typename std::decay<InnerAction>::type> WithoutArgs(
+ InnerAction&& action) {
return {std::forward<InnerAction>(action)};
}
-// Creates an action that returns 'value'. 'value' is passed by value
-// instead of const reference - otherwise Return("string literal")
-// will trigger a compiler error about using array as initializer.
+// Creates an action that returns a value.
+//
+// The returned type can be used with a mock function returning a non-void,
+// non-reference type U as follows:
+//
+// * If R is convertible to U and U is move-constructible, then the action can
+// be used with WillOnce.
+//
+// * If const R& is convertible to U and U is copy-constructible, then the
+// action can be used with both WillOnce and WillRepeatedly.
+//
+// The mock expectation contains the R value from which the U return value is
+// constructed (a move/copy of the argument to Return). This means that the R
+// value will survive at least until the mock object's expectations are cleared
+// or the mock object is destroyed, meaning that U can safely be a
+// reference-like type such as std::string_view:
+//
+// // The mock function returns a view of a copy of the string fed to
+// // Return. The view is valid even after the action is performed.
+// MockFunction<std::string_view()> mock;
+// EXPECT_CALL(mock, Call).WillOnce(Return(std::string("taco")));
+// const std::string_view result = mock.AsStdFunction()();
+// EXPECT_EQ("taco", result);
+//
template <typename R>
internal::ReturnAction<R> Return(R value) {
return internal::ReturnAction<R>(std::move(value));
@@ -1273,6 +1873,8 @@
return internal::ReturnRefOfCopyAction<R>(x);
}
+// DEPRECATED: use Return(x) directly with WillOnce.
+//
// Modifies the parent action (a Return() action) to perform a move of the
// argument instead of a copy.
// Return(ByMove()) actions can only be executed once and will assert this
@@ -1319,7 +1921,7 @@
// Creates an action that sets a pointer referent to a given value.
template <typename T1, typename T2>
-PolymorphicAction<internal::AssignAction<T1, T2> > Assign(T1* ptr, T2 val) {
+PolymorphicAction<internal::AssignAction<T1, T2>> Assign(T1* ptr, T2 val) {
return MakePolymorphicAction(internal::AssignAction<T1, T2>(ptr, val));
}
@@ -1327,8 +1929,8 @@
// Creates an action that sets errno and returns the appropriate error.
template <typename T>
-PolymorphicAction<internal::SetErrnoAndReturnAction<T> >
-SetErrnoAndReturn(int errval, T result) {
+PolymorphicAction<internal::SetErrnoAndReturnAction<T>> SetErrnoAndReturn(
+ int errval, T result) {
return MakePolymorphicAction(
internal::SetErrnoAndReturnAction<T>(errval, result));
}
@@ -1482,7 +2084,8 @@
// Builds an implementation of an Action<> for some particular signature, using
// a class defined by an ACTION* macro.
-template <typename F, typename Impl> struct ActionImpl;
+template <typename F, typename Impl>
+struct ActionImpl;
template <typename Impl>
struct ImplBase {
@@ -1502,7 +2105,7 @@
using args_type = std::tuple<Args...>;
ActionImpl() = default; // Only defined if appropriate for Base.
- explicit ActionImpl(std::shared_ptr<Impl> impl) : Base{std::move(impl)} { }
+ explicit ActionImpl(std::shared_ptr<Impl> impl) : Base{std::move(impl)} {}
R operator()(Args&&... arg) const {
static constexpr size_t kMaxArgs =
@@ -1521,12 +2124,14 @@
// args_type get passed, followed by a dummy of unspecified type for the
// remainder up to 10 explicit args.
static constexpr ExcessiveArg kExcessArg{};
- return static_cast<const Impl&>(*this).template gmock_PerformImpl<
- /*function_type=*/function_type, /*return_type=*/R,
- /*args_type=*/args_type,
- /*argN_type=*/typename std::tuple_element<arg_id, args_type>::type...>(
- /*args=*/args, std::get<arg_id>(args)...,
- ((void)excess_id, kExcessArg)...);
+ return static_cast<const Impl&>(*this)
+ .template gmock_PerformImpl<
+ /*function_type=*/function_type, /*return_type=*/R,
+ /*args_type=*/args_type,
+ /*argN_type=*/
+ typename std::tuple_element<arg_id, args_type>::type...>(
+ /*args=*/args, std::get<arg_id>(args)...,
+ ((void)excess_id, kExcessArg)...);
}
};
@@ -1545,7 +2150,7 @@
#define GMOCK_INTERNAL_ARG_UNUSED(i, data, el) \
, const arg##i##_type& arg##i GTEST_ATTRIBUTE_UNUSED_
-#define GMOCK_ACTION_ARG_TYPES_AND_NAMES_UNUSED_ \
+#define GMOCK_ACTION_ARG_TYPES_AND_NAMES_UNUSED_ \
const args_type& args GTEST_ATTRIBUTE_UNUSED_ GMOCK_PP_REPEAT( \
GMOCK_INTERNAL_ARG_UNUSED, , 10)
@@ -1584,42 +2189,47 @@
#define GMOCK_ACTION_FIELD_PARAMS_(params) \
GMOCK_PP_FOR_EACH(GMOCK_INTERNAL_FIELD_PARAM, , params)
-#define GMOCK_INTERNAL_ACTION(name, full_name, params) \
- template <GMOCK_ACTION_TYPENAME_PARAMS_(params)> \
- class full_name { \
- public: \
- explicit full_name(GMOCK_ACTION_TYPE_GVALUE_PARAMS_(params)) \
- : impl_(std::make_shared<gmock_Impl>( \
- GMOCK_ACTION_GVALUE_PARAMS_(params))) { } \
- full_name(const full_name&) = default; \
- full_name(full_name&&) noexcept = default; \
- template <typename F> \
- operator ::testing::Action<F>() const { \
- return ::testing::internal::MakeAction<F>(impl_); \
- } \
- private: \
- class gmock_Impl { \
- public: \
- explicit gmock_Impl(GMOCK_ACTION_TYPE_GVALUE_PARAMS_(params)) \
- : GMOCK_ACTION_INIT_PARAMS_(params) {} \
- template <typename function_type, typename return_type, \
- typename args_type, GMOCK_ACTION_TEMPLATE_ARGS_NAMES_> \
- return_type gmock_PerformImpl(GMOCK_ACTION_ARG_TYPES_AND_NAMES_) const; \
- GMOCK_ACTION_FIELD_PARAMS_(params) \
- }; \
- std::shared_ptr<const gmock_Impl> impl_; \
- }; \
- template <GMOCK_ACTION_TYPENAME_PARAMS_(params)> \
- inline full_name<GMOCK_ACTION_TYPE_PARAMS_(params)> name( \
- GMOCK_ACTION_TYPE_GVALUE_PARAMS_(params)) { \
- return full_name<GMOCK_ACTION_TYPE_PARAMS_(params)>( \
- GMOCK_ACTION_GVALUE_PARAMS_(params)); \
- } \
- template <GMOCK_ACTION_TYPENAME_PARAMS_(params)> \
- template <typename function_type, typename return_type, typename args_type, \
- GMOCK_ACTION_TEMPLATE_ARGS_NAMES_> \
- return_type full_name<GMOCK_ACTION_TYPE_PARAMS_(params)>::gmock_Impl:: \
- gmock_PerformImpl(GMOCK_ACTION_ARG_TYPES_AND_NAMES_UNUSED_) const
+#define GMOCK_INTERNAL_ACTION(name, full_name, params) \
+ template <GMOCK_ACTION_TYPENAME_PARAMS_(params)> \
+ class full_name { \
+ public: \
+ explicit full_name(GMOCK_ACTION_TYPE_GVALUE_PARAMS_(params)) \
+ : impl_(std::make_shared<gmock_Impl>( \
+ GMOCK_ACTION_GVALUE_PARAMS_(params))) {} \
+ full_name(const full_name&) = default; \
+ full_name(full_name&&) noexcept = default; \
+ template <typename F> \
+ operator ::testing::Action<F>() const { \
+ return ::testing::internal::MakeAction<F>(impl_); \
+ } \
+ \
+ private: \
+ class gmock_Impl { \
+ public: \
+ explicit gmock_Impl(GMOCK_ACTION_TYPE_GVALUE_PARAMS_(params)) \
+ : GMOCK_ACTION_INIT_PARAMS_(params) {} \
+ template <typename function_type, typename return_type, \
+ typename args_type, GMOCK_ACTION_TEMPLATE_ARGS_NAMES_> \
+ return_type gmock_PerformImpl(GMOCK_ACTION_ARG_TYPES_AND_NAMES_) const; \
+ GMOCK_ACTION_FIELD_PARAMS_(params) \
+ }; \
+ std::shared_ptr<const gmock_Impl> impl_; \
+ }; \
+ template <GMOCK_ACTION_TYPENAME_PARAMS_(params)> \
+ inline full_name<GMOCK_ACTION_TYPE_PARAMS_(params)> name( \
+ GMOCK_ACTION_TYPE_GVALUE_PARAMS_(params)) GTEST_MUST_USE_RESULT_; \
+ template <GMOCK_ACTION_TYPENAME_PARAMS_(params)> \
+ inline full_name<GMOCK_ACTION_TYPE_PARAMS_(params)> name( \
+ GMOCK_ACTION_TYPE_GVALUE_PARAMS_(params)) { \
+ return full_name<GMOCK_ACTION_TYPE_PARAMS_(params)>( \
+ GMOCK_ACTION_GVALUE_PARAMS_(params)); \
+ } \
+ template <GMOCK_ACTION_TYPENAME_PARAMS_(params)> \
+ template <typename function_type, typename return_type, typename args_type, \
+ GMOCK_ACTION_TEMPLATE_ARGS_NAMES_> \
+ return_type \
+ full_name<GMOCK_ACTION_TYPE_PARAMS_(params)>::gmock_Impl::gmock_PerformImpl( \
+ GMOCK_ACTION_ARG_TYPES_AND_NAMES_UNUSED_) const
} // namespace internal
@@ -1627,12 +2237,13 @@
#define ACTION(name) \
class name##Action { \
public: \
- explicit name##Action() noexcept {} \
- name##Action(const name##Action&) noexcept {} \
+ explicit name##Action() noexcept {} \
+ name##Action(const name##Action&) noexcept {} \
template <typename F> \
operator ::testing::Action<F>() const { \
return ::testing::internal::MakeAction<F, gmock_Impl>(); \
} \
+ \
private: \
class gmock_Impl { \
public: \
@@ -1681,7 +2292,7 @@
} // namespace testing
#ifdef _MSC_VER
-# pragma warning(pop)
+#pragma warning(pop)
#endif
#endif // GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_ACTIONS_H_
diff --git a/third_party/googletest/src/googlemock/include/gmock/gmock-cardinalities.h b/third_party/googletest/src/googlemock/include/gmock/gmock-cardinalities.h
index fc7f803..b6ab648 100644
--- a/third_party/googletest/src/googlemock/include/gmock/gmock-cardinalities.h
+++ b/third_party/googletest/src/googlemock/include/gmock/gmock-cardinalities.h
@@ -27,21 +27,23 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Google Mock - a framework for writing C++ mock classes.
//
// This file implements some commonly used cardinalities. More
// cardinalities can be defined by the user implementing the
// CardinalityInterface interface if necessary.
-// GOOGLETEST_CM0002 DO NOT DELETE
+// IWYU pragma: private, include "gmock/gmock.h"
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_CARDINALITIES_H_
#define GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_CARDINALITIES_H_
#include <limits.h>
+
#include <memory>
#include <ostream> // NOLINT
+
#include "gmock/internal/gmock-port.h"
#include "gtest/gtest.h"
@@ -116,7 +118,7 @@
// cardinality, i.e. exceed the maximum number of allowed calls.
bool IsOverSaturatedByCallCount(int call_count) const {
return impl_->IsSaturatedByCallCount(call_count) &&
- !impl_->IsSatisfiedByCallCount(call_count);
+ !impl_->IsSatisfiedByCallCount(call_count);
}
// Describes self to an ostream
diff --git a/third_party/googletest/src/googlemock/include/gmock/gmock-function-mocker.h b/third_party/googletest/src/googlemock/include/gmock/gmock-function-mocker.h
index 0fc6f6f..f565d98 100644
--- a/third_party/googletest/src/googlemock/include/gmock/gmock-function-mocker.h
+++ b/third_party/googletest/src/googlemock/include/gmock/gmock-function-mocker.h
@@ -31,7 +31,8 @@
//
// This file implements MOCK_METHOD.
-// GOOGLETEST_CM0002 DO NOT DELETE
+// IWYU pragma: private, include "gmock/gmock.h"
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLEMOCK_INCLUDE_GMOCK_INTERNAL_GMOCK_FUNCTION_MOCKER_H_ // NOLINT
#define GOOGLEMOCK_INCLUDE_GMOCK_INTERNAL_GMOCK_FUNCTION_MOCKER_H_ // NOLINT
@@ -64,6 +65,39 @@
}
};
+constexpr bool PrefixOf(const char* a, const char* b) {
+ return *a == 0 || (*a == *b && internal::PrefixOf(a + 1, b + 1));
+}
+
+template <int N, int M>
+constexpr bool StartsWith(const char (&prefix)[N], const char (&str)[M]) {
+ return N <= M && internal::PrefixOf(prefix, str);
+}
+
+template <int N, int M>
+constexpr bool EndsWith(const char (&suffix)[N], const char (&str)[M]) {
+ return N <= M && internal::PrefixOf(suffix, str + M - N);
+}
+
+template <int N, int M>
+constexpr bool Equals(const char (&a)[N], const char (&b)[M]) {
+ return N == M && internal::PrefixOf(a, b);
+}
+
+template <int N>
+constexpr bool ValidateSpec(const char (&spec)[N]) {
+ return internal::Equals("const", spec) ||
+ internal::Equals("override", spec) ||
+ internal::Equals("final", spec) ||
+ internal::Equals("noexcept", spec) ||
+ (internal::StartsWith("noexcept(", spec) &&
+ internal::EndsWith(")", spec)) ||
+ internal::Equals("ref(&)", spec) ||
+ internal::Equals("ref(&&)", spec) ||
+ (internal::StartsWith("Calltype(", spec) &&
+ internal::EndsWith(")", spec));
+}
+
} // namespace internal
// The style guide prohibits "using" statements in a namespace scope
@@ -86,17 +120,18 @@
#define GMOCK_INTERNAL_MOCK_METHOD_ARG_3(_Ret, _MethodName, _Args) \
GMOCK_INTERNAL_MOCK_METHOD_ARG_4(_Ret, _MethodName, _Args, ())
-#define GMOCK_INTERNAL_MOCK_METHOD_ARG_4(_Ret, _MethodName, _Args, _Spec) \
- GMOCK_INTERNAL_ASSERT_PARENTHESIS(_Args); \
- GMOCK_INTERNAL_ASSERT_PARENTHESIS(_Spec); \
- GMOCK_INTERNAL_ASSERT_VALID_SIGNATURE( \
- GMOCK_PP_NARG0 _Args, GMOCK_INTERNAL_SIGNATURE(_Ret, _Args)); \
- GMOCK_INTERNAL_ASSERT_VALID_SPEC(_Spec) \
- GMOCK_INTERNAL_MOCK_METHOD_IMPL( \
- GMOCK_PP_NARG0 _Args, _MethodName, GMOCK_INTERNAL_HAS_CONST(_Spec), \
- GMOCK_INTERNAL_HAS_OVERRIDE(_Spec), GMOCK_INTERNAL_HAS_FINAL(_Spec), \
- GMOCK_INTERNAL_GET_NOEXCEPT_SPEC(_Spec), \
- GMOCK_INTERNAL_GET_CALLTYPE(_Spec), GMOCK_INTERNAL_GET_REF_SPEC(_Spec), \
+#define GMOCK_INTERNAL_MOCK_METHOD_ARG_4(_Ret, _MethodName, _Args, _Spec) \
+ GMOCK_INTERNAL_ASSERT_PARENTHESIS(_Args); \
+ GMOCK_INTERNAL_ASSERT_PARENTHESIS(_Spec); \
+ GMOCK_INTERNAL_ASSERT_VALID_SIGNATURE( \
+ GMOCK_PP_NARG0 _Args, GMOCK_INTERNAL_SIGNATURE(_Ret, _Args)); \
+ GMOCK_INTERNAL_ASSERT_VALID_SPEC(_Spec) \
+ GMOCK_INTERNAL_MOCK_METHOD_IMPL( \
+ GMOCK_PP_NARG0 _Args, _MethodName, GMOCK_INTERNAL_HAS_CONST(_Spec), \
+ GMOCK_INTERNAL_HAS_OVERRIDE(_Spec), GMOCK_INTERNAL_HAS_FINAL(_Spec), \
+ GMOCK_INTERNAL_GET_NOEXCEPT_SPEC(_Spec), \
+ GMOCK_INTERNAL_GET_CALLTYPE_SPEC(_Spec), \
+ GMOCK_INTERNAL_GET_REF_SPEC(_Spec), \
(GMOCK_INTERNAL_SIGNATURE(_Ret, _Args)))
#define GMOCK_INTERNAL_MOCK_METHOD_ARG_5(...) \
@@ -166,11 +201,11 @@
GMOCK_INTERNAL_A_MATCHER_ARGUMENT, _Signature, _N)); \
} \
mutable ::testing::FunctionMocker<GMOCK_PP_REMOVE_PARENS(_Signature)> \
- GMOCK_MOCKER_(_N, _Constness, _MethodName)
+ GMOCK_MOCKER_(_N, _Constness, _MethodName)
#define GMOCK_INTERNAL_EXPAND(...) __VA_ARGS__
-// Five Valid modifiers.
+// Valid modifiers.
#define GMOCK_INTERNAL_HAS_CONST(_Tuple) \
GMOCK_PP_HAS_COMMA(GMOCK_PP_FOR_EACH(GMOCK_INTERNAL_DETECT_CONST, ~, _Tuple))
@@ -189,6 +224,14 @@
GMOCK_PP_HAS_COMMA(GMOCK_INTERNAL_DETECT_NOEXCEPT(_i, _, _elem)), \
_elem, )
+#define GMOCK_INTERNAL_GET_CALLTYPE_SPEC(_Tuple) \
+ GMOCK_PP_FOR_EACH(GMOCK_INTERNAL_CALLTYPE_SPEC_IF_CALLTYPE, ~, _Tuple)
+
+#define GMOCK_INTERNAL_CALLTYPE_SPEC_IF_CALLTYPE(_i, _, _elem) \
+ GMOCK_PP_IF( \
+ GMOCK_PP_HAS_COMMA(GMOCK_INTERNAL_DETECT_CALLTYPE(_i, _, _elem)), \
+ GMOCK_PP_CAT(GMOCK_INTERNAL_UNPACK_, _elem), )
+
#define GMOCK_INTERNAL_GET_REF_SPEC(_Tuple) \
GMOCK_PP_FOR_EACH(GMOCK_INTERNAL_REF_SPEC_IF_REF, ~, _Tuple)
@@ -196,19 +239,25 @@
GMOCK_PP_IF(GMOCK_PP_HAS_COMMA(GMOCK_INTERNAL_DETECT_REF(_i, _, _elem)), \
GMOCK_PP_CAT(GMOCK_INTERNAL_UNPACK_, _elem), )
-#define GMOCK_INTERNAL_GET_CALLTYPE(_Tuple) \
- GMOCK_PP_FOR_EACH(GMOCK_INTERNAL_GET_CALLTYPE_IMPL, ~, _Tuple)
-
-#define GMOCK_INTERNAL_ASSERT_VALID_SPEC_ELEMENT(_i, _, _elem) \
- static_assert( \
- (GMOCK_PP_HAS_COMMA(GMOCK_INTERNAL_DETECT_CONST(_i, _, _elem)) + \
- GMOCK_PP_HAS_COMMA(GMOCK_INTERNAL_DETECT_OVERRIDE(_i, _, _elem)) + \
- GMOCK_PP_HAS_COMMA(GMOCK_INTERNAL_DETECT_FINAL(_i, _, _elem)) + \
- GMOCK_PP_HAS_COMMA(GMOCK_INTERNAL_DETECT_NOEXCEPT(_i, _, _elem)) + \
- GMOCK_PP_HAS_COMMA(GMOCK_INTERNAL_DETECT_REF(_i, _, _elem)) + \
- GMOCK_INTERNAL_IS_CALLTYPE(_elem)) == 1, \
- GMOCK_PP_STRINGIZE( \
+#ifdef GMOCK_INTERNAL_STRICT_SPEC_ASSERT
+#define GMOCK_INTERNAL_ASSERT_VALID_SPEC_ELEMENT(_i, _, _elem) \
+ static_assert( \
+ ::testing::internal::ValidateSpec(GMOCK_PP_STRINGIZE(_elem)), \
+ "Token \'" GMOCK_PP_STRINGIZE( \
+ _elem) "\' cannot be recognized as a valid specification " \
+ "modifier. Is a ',' missing?");
+#else
+#define GMOCK_INTERNAL_ASSERT_VALID_SPEC_ELEMENT(_i, _, _elem) \
+ static_assert( \
+ (GMOCK_PP_HAS_COMMA(GMOCK_INTERNAL_DETECT_CONST(_i, _, _elem)) + \
+ GMOCK_PP_HAS_COMMA(GMOCK_INTERNAL_DETECT_OVERRIDE(_i, _, _elem)) + \
+ GMOCK_PP_HAS_COMMA(GMOCK_INTERNAL_DETECT_FINAL(_i, _, _elem)) + \
+ GMOCK_PP_HAS_COMMA(GMOCK_INTERNAL_DETECT_NOEXCEPT(_i, _, _elem)) + \
+ GMOCK_PP_HAS_COMMA(GMOCK_INTERNAL_DETECT_REF(_i, _, _elem)) + \
+ GMOCK_PP_HAS_COMMA(GMOCK_INTERNAL_DETECT_CALLTYPE(_i, _, _elem))) == 1, \
+ GMOCK_PP_STRINGIZE( \
_elem) " cannot be recognized as a valid specification modifier.");
+#endif // GMOCK_INTERNAL_STRICT_SPEC_ASSERT
// Modifiers implementation.
#define GMOCK_INTERNAL_DETECT_CONST(_i, _, _elem) \
@@ -238,26 +287,12 @@
#define GMOCK_INTERNAL_UNPACK_ref(x) x
-#define GMOCK_INTERNAL_GET_CALLTYPE_IMPL(_i, _, _elem) \
- GMOCK_PP_IF(GMOCK_INTERNAL_IS_CALLTYPE(_elem), \
- GMOCK_INTERNAL_GET_VALUE_CALLTYPE, GMOCK_PP_EMPTY) \
- (_elem)
+#define GMOCK_INTERNAL_DETECT_CALLTYPE(_i, _, _elem) \
+ GMOCK_PP_CAT(GMOCK_INTERNAL_DETECT_CALLTYPE_I_, _elem)
-// TODO(iserna): GMOCK_INTERNAL_IS_CALLTYPE and
-// GMOCK_INTERNAL_GET_VALUE_CALLTYPE needed more expansions to work on windows
-// maybe they can be simplified somehow.
-#define GMOCK_INTERNAL_IS_CALLTYPE(_arg) \
- GMOCK_INTERNAL_IS_CALLTYPE_I( \
- GMOCK_PP_CAT(GMOCK_INTERNAL_IS_CALLTYPE_HELPER_, _arg))
-#define GMOCK_INTERNAL_IS_CALLTYPE_I(_arg) GMOCK_PP_IS_ENCLOSED_PARENS(_arg)
+#define GMOCK_INTERNAL_DETECT_CALLTYPE_I_Calltype ,
-#define GMOCK_INTERNAL_GET_VALUE_CALLTYPE(_arg) \
- GMOCK_INTERNAL_GET_VALUE_CALLTYPE_I( \
- GMOCK_PP_CAT(GMOCK_INTERNAL_IS_CALLTYPE_HELPER_, _arg))
-#define GMOCK_INTERNAL_GET_VALUE_CALLTYPE_I(_arg) \
- GMOCK_PP_IDENTITY _arg
-
-#define GMOCK_INTERNAL_IS_CALLTYPE_HELPER_Calltype
+#define GMOCK_INTERNAL_UNPACK_Calltype(...) __VA_ARGS__
// Note: The use of `identity_t` here allows _Ret to represent return types that
// would normally need to be specified in a different way. For example, a method
diff --git a/third_party/googletest/src/googlemock/include/gmock/gmock-matchers.h b/third_party/googletest/src/googlemock/include/gmock/gmock-matchers.h
index 86be9c1..6282901 100644
--- a/third_party/googletest/src/googlemock/include/gmock/gmock-matchers.h
+++ b/third_party/googletest/src/googlemock/include/gmock/gmock-matchers.h
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Google Mock - a framework for writing C++ mock classes.
//
// The MATCHER* family of macros can be used in a namespace scope to
@@ -250,7 +249,8 @@
// See googletest/include/gtest/gtest-matchers.h for the definition of class
// Matcher, class MatcherInterface, and others.
-// GOOGLETEST_CM0002 DO NOT DELETE
+// IWYU pragma: private, include "gmock/gmock.h"
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_MATCHERS_H_
#define GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_MATCHERS_H_
@@ -313,7 +313,9 @@
private:
::std::stringstream ss_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(StringMatchResultListener);
+ StringMatchResultListener(const StringMatchResultListener&) = delete;
+ StringMatchResultListener& operator=(const StringMatchResultListener&) =
+ delete;
};
// Anything inside the 'internal' namespace IS INTERNAL IMPLEMENTATION
@@ -396,7 +398,7 @@
// is already a Matcher. This only compiles when type T can be
// statically converted to type U.
template <typename T, typename U>
-class MatcherCastImpl<T, Matcher<U> > {
+class MatcherCastImpl<T, Matcher<U>> {
public:
static Matcher<T> Cast(const Matcher<U>& source_matcher) {
return Matcher<T>(new Impl(source_matcher));
@@ -450,7 +452,7 @@
// This even more specialized version is used for efficiently casting
// a matcher to its own type.
template <typename T>
-class MatcherCastImpl<T, Matcher<T> > {
+class MatcherCastImpl<T, Matcher<T>> {
public:
static Matcher<T> Cast(const Matcher<T>& matcher) { return matcher; }
};
@@ -533,19 +535,18 @@
"T must be implicitly convertible to U");
// Enforce that we are not converting a non-reference type T to a reference
// type U.
- GTEST_COMPILE_ASSERT_(
- std::is_reference<T>::value || !std::is_reference<U>::value,
- cannot_convert_non_reference_arg_to_reference);
+ static_assert(std::is_reference<T>::value || !std::is_reference<U>::value,
+ "cannot convert non reference arg to reference");
// In case both T and U are arithmetic types, enforce that the
// conversion is not lossy.
typedef GTEST_REMOVE_REFERENCE_AND_CONST_(T) RawT;
typedef GTEST_REMOVE_REFERENCE_AND_CONST_(U) RawU;
constexpr bool kTIsOther = GMOCK_KIND_OF_(RawT) == internal::kOther;
constexpr bool kUIsOther = GMOCK_KIND_OF_(RawU) == internal::kOther;
- GTEST_COMPILE_ASSERT_(
+ static_assert(
kTIsOther || kUIsOther ||
- (internal::LosslessArithmeticConvertible<RawT, RawU>::value),
- conversion_of_arithmetic_types_must_be_lossless);
+ (internal::LosslessArithmeticConvertible<RawT, RawU>::value),
+ "conversion of arithmetic types must be lossless");
return MatcherCast<T>(matcher);
}
@@ -678,9 +679,9 @@
const ValueTuple& value_tuple) {
// Makes sure that matcher_tuple and value_tuple have the same
// number of fields.
- GTEST_COMPILE_ASSERT_(std::tuple_size<MatcherTuple>::value ==
- std::tuple_size<ValueTuple>::value,
- matcher_and_value_have_different_numbers_of_fields);
+ static_assert(std::tuple_size<MatcherTuple>::value ==
+ std::tuple_size<ValueTuple>::value,
+ "matcher and value have different numbers of fields");
return TuplePrefix<std::tuple_size<ValueTuple>::value>::Matches(matcher_tuple,
value_tuple);
}
@@ -689,8 +690,7 @@
// is no failure, nothing will be streamed to os.
template <typename MatcherTuple, typename ValueTuple>
void ExplainMatchFailureTupleTo(const MatcherTuple& matchers,
- const ValueTuple& values,
- ::std::ostream* os) {
+ const ValueTuple& values, ::std::ostream* os) {
TuplePrefix<std::tuple_size<MatcherTuple>::value>::ExplainMatchFailuresTo(
matchers, values, os);
}
@@ -714,14 +714,14 @@
private:
template <typename Tup, size_t kRemainingSize>
struct IterateOverTuple {
- OutIter operator() (Func f, const Tup& t, OutIter out) const {
+ OutIter operator()(Func f, const Tup& t, OutIter out) const {
*out++ = f(::std::get<TupleSize::value - kRemainingSize>(t));
return IterateOverTuple<Tup, kRemainingSize - 1>()(f, t, out);
}
};
template <typename Tup>
struct IterateOverTuple<Tup, 0> {
- OutIter operator() (Func /* f */, const Tup& /* t */, OutIter out) const {
+ OutIter operator()(Func /* f */, const Tup& /* t */, OutIter out) const {
return out;
}
};
@@ -767,9 +767,7 @@
}
void DescribeTo(::std::ostream* os) const { *os << "is NULL"; }
- void DescribeNegationTo(::std::ostream* os) const {
- *os << "isn't NULL";
- }
+ void DescribeNegationTo(::std::ostream* os) const { *os << "isn't NULL"; }
};
// Implements the polymorphic NotNull() matcher, which matches any raw or smart
@@ -783,9 +781,7 @@
}
void DescribeTo(::std::ostream* os) const { *os << "isn't NULL"; }
- void DescribeNegationTo(::std::ostream* os) const {
- *os << "is NULL";
- }
+ void DescribeNegationTo(::std::ostream* os) const { *os << "is NULL"; }
};
// Ref(variable) matches any argument that is a reference to
@@ -871,8 +867,7 @@
// String comparison for narrow or wide strings that can have embedded NUL
// characters.
template <typename StringType>
-bool CaseInsensitiveStringEquals(const StringType& s1,
- const StringType& s2) {
+bool CaseInsensitiveStringEquals(const StringType& s1, const StringType& s2) {
// Are the heads equal?
if (!CaseInsensitiveCStringEquals(s1.c_str(), s2.c_str())) {
return false;
@@ -933,8 +928,8 @@
bool MatchAndExplain(const MatcheeStringType& s,
MatchResultListener* /* listener */) const {
const StringType s2(s);
- const bool eq = case_sensitive_ ? s2 == string_ :
- CaseInsensitiveStringEquals(s2, string_);
+ const bool eq = case_sensitive_ ? s2 == string_
+ : CaseInsensitiveStringEquals(s2, string_);
return expect_eq_ == eq;
}
@@ -1021,8 +1016,7 @@
template <typename StringType>
class StartsWithMatcher {
public:
- explicit StartsWithMatcher(const StringType& prefix) : prefix_(prefix) {
- }
+ explicit StartsWithMatcher(const StringType& prefix) : prefix_(prefix) {}
#if GTEST_INTERNAL_HAS_STRING_VIEW
bool MatchAndExplain(const internal::StringView& s,
@@ -1053,7 +1047,7 @@
MatchResultListener* /* listener */) const {
const StringType& s2(s);
return s2.length() >= prefix_.length() &&
- s2.substr(0, prefix_.length()) == prefix_;
+ s2.substr(0, prefix_.length()) == prefix_;
}
void DescribeTo(::std::ostream* os) const {
@@ -1107,7 +1101,7 @@
MatchResultListener* /* listener */) const {
const StringType& s2(s);
return s2.length() >= suffix_.length() &&
- s2.substr(s2.length() - suffix_.length()) == suffix_;
+ s2.substr(s2.length() - suffix_.length()) == suffix_;
}
void DescribeTo(::std::ostream* os) const {
@@ -1124,6 +1118,45 @@
const StringType suffix_;
};
+// Implements the polymorphic WhenBase64Unescaped(matcher) matcher, which can be
+// used as a Matcher<T> as long as T can be converted to a string.
+class WhenBase64UnescapedMatcher {
+ public:
+ using is_gtest_matcher = void;
+
+ explicit WhenBase64UnescapedMatcher(
+ const Matcher<const std::string&>& internal_matcher)
+ : internal_matcher_(internal_matcher) {}
+
+ // Matches anything that can convert to std::string.
+ template <typename MatcheeStringType>
+ bool MatchAndExplain(const MatcheeStringType& s,
+ MatchResultListener* listener) const {
+ const std::string s2(s); // NOLINT (needed for working with string_view).
+ std::string unescaped;
+ if (!internal::Base64Unescape(s2, &unescaped)) {
+ if (listener != nullptr) {
+ *listener << "is not a valid base64 escaped string";
+ }
+ return false;
+ }
+ return MatchPrintAndExplain(unescaped, internal_matcher_, listener);
+ }
+
+ void DescribeTo(::std::ostream* os) const {
+ *os << "matches after Base64Unescape ";
+ internal_matcher_.DescribeTo(os);
+ }
+
+ void DescribeNegationTo(::std::ostream* os) const {
+ *os << "does not match after Base64Unescape ";
+ internal_matcher_.DescribeTo(os);
+ }
+
+ private:
+ const Matcher<const std::string&> internal_matcher_;
+};
+
// Implements a matcher that compares the two fields of a 2-tuple
// using one of the ==, <=, <, etc, operators. The two fields being
// compared don't have to have the same type.
@@ -1197,8 +1230,7 @@
template <typename T>
class NotMatcherImpl : public MatcherInterface<const T&> {
public:
- explicit NotMatcherImpl(const Matcher<T>& matcher)
- : matcher_(matcher) {}
+ explicit NotMatcherImpl(const Matcher<T>& matcher) : matcher_(matcher) {}
bool MatchAndExplain(const T& x,
MatchResultListener* listener) const override {
@@ -1242,7 +1274,7 @@
template <typename T>
class AllOfMatcherImpl : public MatcherInterface<const T&> {
public:
- explicit AllOfMatcherImpl(std::vector<Matcher<T> > matchers)
+ explicit AllOfMatcherImpl(std::vector<Matcher<T>> matchers)
: matchers_(std::move(matchers)) {}
void DescribeTo(::std::ostream* os) const override {
@@ -1293,7 +1325,7 @@
}
private:
- const std::vector<Matcher<T> > matchers_;
+ const std::vector<Matcher<T>> matchers_;
};
// VariadicMatcher is used for the variadic implementation of
@@ -1316,14 +1348,14 @@
// all of the provided matchers (Matcher1, Matcher2, ...) can match.
template <typename T>
operator Matcher<T>() const {
- std::vector<Matcher<T> > values;
+ std::vector<Matcher<T>> values;
CreateVariadicMatcher<T>(&values, std::integral_constant<size_t, 0>());
return Matcher<T>(new CombiningMatcher<T>(std::move(values)));
}
private:
template <typename T, size_t I>
- void CreateVariadicMatcher(std::vector<Matcher<T> >* values,
+ void CreateVariadicMatcher(std::vector<Matcher<T>>* values,
std::integral_constant<size_t, I>) const {
values->push_back(SafeMatcherCast<T>(std::get<I>(matchers_)));
CreateVariadicMatcher<T>(values, std::integral_constant<size_t, I + 1>());
@@ -1331,7 +1363,7 @@
template <typename T>
void CreateVariadicMatcher(
- std::vector<Matcher<T> >*,
+ std::vector<Matcher<T>>*,
std::integral_constant<size_t, sizeof...(Args)>) const {}
std::tuple<Args...> matchers_;
@@ -1347,7 +1379,7 @@
template <typename T>
class AnyOfMatcherImpl : public MatcherInterface<const T&> {
public:
- explicit AnyOfMatcherImpl(std::vector<Matcher<T> > matchers)
+ explicit AnyOfMatcherImpl(std::vector<Matcher<T>> matchers)
: matchers_(std::move(matchers)) {}
void DescribeTo(::std::ostream* os) const override {
@@ -1398,13 +1430,35 @@
}
private:
- const std::vector<Matcher<T> > matchers_;
+ const std::vector<Matcher<T>> matchers_;
};
// AnyOfMatcher is used for the variadic implementation of AnyOf(m_1, m_2, ...).
template <typename... Args>
using AnyOfMatcher = VariadicMatcher<AnyOfMatcherImpl, Args...>;
+// ConditionalMatcher is the implementation of Conditional(cond, m1, m2)
+template <typename MatcherTrue, typename MatcherFalse>
+class ConditionalMatcher {
+ public:
+ ConditionalMatcher(bool condition, MatcherTrue matcher_true,
+ MatcherFalse matcher_false)
+ : condition_(condition),
+ matcher_true_(std::move(matcher_true)),
+ matcher_false_(std::move(matcher_false)) {}
+
+ template <typename T>
+ operator Matcher<T>() const { // NOLINT(runtime/explicit)
+ return condition_ ? SafeMatcherCast<T>(matcher_true_)
+ : SafeMatcherCast<T>(matcher_false_);
+ }
+
+ private:
+ bool condition_;
+ MatcherTrue matcher_true_;
+ MatcherFalse matcher_false_;
+};
+
// Wrapper for implementation of Any/AllOfArray().
template <template <class> class MatcherImpl, typename T>
class SomeOfArrayMatcher {
@@ -1454,8 +1508,7 @@
// We cannot write 'return !!predicate_(x);' as that doesn't work
// when predicate_(x) returns a class convertible to bool but
// having no operator!().
- if (predicate_(x))
- return true;
+ if (predicate_(x)) return true;
*listener << "didn't satisfy the given predicate";
return false;
}
@@ -1563,8 +1616,8 @@
// used for implementing ASSERT_THAT() and EXPECT_THAT().
// Implementation detail: 'matcher' is received by-value to force decaying.
template <typename M>
-inline PredicateFormatterFromMatcher<M>
-MakePredicateFormatterFromMatcher(M matcher) {
+inline PredicateFormatterFromMatcher<M> MakePredicateFormatterFromMatcher(
+ M matcher) {
return PredicateFormatterFromMatcher<M>(std::move(matcher));
}
@@ -1579,9 +1632,7 @@
}
void DescribeTo(::std::ostream* os) const { *os << "is NaN"; }
- void DescribeNegationTo(::std::ostream* os) const {
- *os << "isn't NaN";
- }
+ void DescribeNegationTo(::std::ostream* os) const { *os << "isn't NaN"; }
};
// Implements the polymorphic floating point equality matcher, which matches
@@ -1597,9 +1648,8 @@
// equality comparisons between NANs will always return false. We specify a
// negative max_abs_error_ term to indicate that ULP-based approximation will
// be used for comparison.
- FloatingEqMatcher(FloatType expected, bool nan_eq_nan) :
- expected_(expected), nan_eq_nan_(nan_eq_nan), max_abs_error_(-1) {
- }
+ FloatingEqMatcher(FloatType expected, bool nan_eq_nan)
+ : expected_(expected), nan_eq_nan_(nan_eq_nan), max_abs_error_(-1) {}
// Constructor that supports a user-specified max_abs_error that will be used
// for comparison instead of ULP-based approximation. The max absolute
@@ -1661,8 +1711,8 @@
// os->precision() returns the previously set precision, which we
// store to restore the ostream to its original configuration
// after outputting.
- const ::std::streamsize old_precision = os->precision(
- ::std::numeric_limits<FloatType>::digits10 + 2);
+ const ::std::streamsize old_precision =
+ os->precision(::std::numeric_limits<FloatType>::digits10 + 2);
if (FloatingPoint<FloatType>(expected_).is_nan()) {
if (nan_eq_nan_) {
*os << "is NaN";
@@ -1680,8 +1730,8 @@
void DescribeNegationTo(::std::ostream* os) const override {
// As before, get original precision.
- const ::std::streamsize old_precision = os->precision(
- ::std::numeric_limits<FloatType>::digits10 + 2);
+ const ::std::streamsize old_precision =
+ os->precision(::std::numeric_limits<FloatType>::digits10 + 2);
if (FloatingPoint<FloatType>(expected_).is_nan()) {
if (nan_eq_nan_) {
*os << "isn't NaN";
@@ -1699,9 +1749,7 @@
}
private:
- bool HasMaxAbsError() const {
- return max_abs_error_ >= 0;
- }
+ bool HasMaxAbsError() const { return max_abs_error_ >= 0; }
const FloatType expected_;
const bool nan_eq_nan_;
@@ -1773,9 +1821,8 @@
template <typename Tuple>
class Impl : public MatcherInterface<Tuple> {
public:
- Impl(FloatType max_abs_error, bool nan_eq_nan) :
- max_abs_error_(max_abs_error),
- nan_eq_nan_(nan_eq_nan) {}
+ Impl(FloatType max_abs_error, bool nan_eq_nan)
+ : max_abs_error_(max_abs_error), nan_eq_nan_(nan_eq_nan) {}
bool MatchAndExplain(Tuple args,
MatchResultListener* listener) const override {
@@ -1951,9 +1998,7 @@
protected:
const Matcher<To> matcher_;
- static std::string GetToName() {
- return GetTypeName<To>();
- }
+ static std::string GetToName() { return GetTypeName<To>(); }
private:
static void GetCastTypeDescription(::std::ostream* os) {
@@ -2090,7 +2135,7 @@
}
template <typename T>
- bool MatchAndExplain(const T&value, MatchResultListener* listener) const {
+ bool MatchAndExplain(const T& value, MatchResultListener* listener) const {
return MatchAndExplainImpl(
typename std::is_pointer<typename std::remove_const<T>::type>::type(),
value, listener);
@@ -2142,16 +2187,16 @@
// Specialization for function pointers.
template <typename ArgType, typename ResType>
-struct CallableTraits<ResType(*)(ArgType)> {
+struct CallableTraits<ResType (*)(ArgType)> {
typedef ResType ResultType;
- typedef ResType(*StorageType)(ArgType);
+ typedef ResType (*StorageType)(ArgType);
- static void CheckIsValid(ResType(*f)(ArgType)) {
+ static void CheckIsValid(ResType (*f)(ArgType)) {
GTEST_CHECK_(f != nullptr)
<< "NULL function pointer is passed into ResultOf().";
}
template <typename T>
- static ResType Invoke(ResType(*f)(ArgType), T arg) {
+ static ResType Invoke(ResType (*f)(ArgType), T arg) {
return (*f)(arg);
}
};
@@ -2162,13 +2207,21 @@
class ResultOfMatcher {
public:
ResultOfMatcher(Callable callable, InnerMatcher matcher)
- : callable_(std::move(callable)), matcher_(std::move(matcher)) {
+ : ResultOfMatcher(/*result_description=*/"", std::move(callable),
+ std::move(matcher)) {}
+
+ ResultOfMatcher(const std::string& result_description, Callable callable,
+ InnerMatcher matcher)
+ : result_description_(result_description),
+ callable_(std::move(callable)),
+ matcher_(std::move(matcher)) {
CallableTraits<Callable>::CheckIsValid(callable_);
}
template <typename T>
operator Matcher<T>() const {
- return Matcher<T>(new Impl<const T&>(callable_, matcher_));
+ return Matcher<T>(
+ new Impl<const T&>(result_description_, callable_, matcher_));
}
private:
@@ -2181,21 +2234,36 @@
public:
template <typename M>
- Impl(const CallableStorageType& callable, const M& matcher)
- : callable_(callable), matcher_(MatcherCast<ResultType>(matcher)) {}
+ Impl(const std::string& result_description,
+ const CallableStorageType& callable, const M& matcher)
+ : result_description_(result_description),
+ callable_(callable),
+ matcher_(MatcherCast<ResultType>(matcher)) {}
void DescribeTo(::std::ostream* os) const override {
- *os << "is mapped by the given callable to a value that ";
+ if (result_description_.empty()) {
+ *os << "is mapped by the given callable to a value that ";
+ } else {
+ *os << "whose " << result_description_ << " ";
+ }
matcher_.DescribeTo(os);
}
void DescribeNegationTo(::std::ostream* os) const override {
- *os << "is mapped by the given callable to a value that ";
+ if (result_description_.empty()) {
+ *os << "is mapped by the given callable to a value that ";
+ } else {
+ *os << "whose " << result_description_ << " ";
+ }
matcher_.DescribeNegationTo(os);
}
bool MatchAndExplain(T obj, MatchResultListener* listener) const override {
- *listener << "which is mapped by the given callable to ";
+ if (result_description_.empty()) {
+ *listener << "which is mapped by the given callable to ";
+ } else {
+ *listener << "whose " << result_description_ << " is ";
+ }
// Cannot pass the return value directly to MatchPrintAndExplain, which
// takes a non-const reference as argument.
// Also, specifying template argument explicitly is needed because T could
@@ -2206,6 +2274,7 @@
}
private:
+ const std::string result_description_;
// Functors often define operator() as non-const method even though
// they are actually stateless. But we need to use them even when
// 'this' is a const pointer. It's the user's responsibility not to
@@ -2215,6 +2284,7 @@
const Matcher<ResultType> matcher_;
}; // class Impl
+ const std::string result_description_;
const CallableStorageType callable_;
const InnerMatcher matcher_;
};
@@ -2224,8 +2294,7 @@
class SizeIsMatcher {
public:
explicit SizeIsMatcher(const SizeMatcher& size_matcher)
- : size_matcher_(size_matcher) {
- }
+ : size_matcher_(size_matcher) {}
template <typename Container>
operator Matcher<Container>() const {
@@ -2253,8 +2322,8 @@
SizeType size = container.size();
StringMatchResultListener size_listener;
const bool result = size_matcher_.MatchAndExplain(size, &size_listener);
- *listener
- << "whose size " << size << (result ? " matches" : " doesn't match");
+ *listener << "whose size " << size
+ << (result ? " matches" : " doesn't match");
PrintIfNotEmpty(size_listener.str(), listener->stream());
return result;
}
@@ -2283,8 +2352,9 @@
template <typename Container>
class Impl : public MatcherInterface<Container> {
public:
- typedef internal::StlContainerView<
- GTEST_REMOVE_REFERENCE_AND_CONST_(Container)> ContainerView;
+ typedef internal::StlContainerView<GTEST_REMOVE_REFERENCE_AND_CONST_(
+ Container)>
+ ContainerView;
typedef typename std::iterator_traits<
typename ContainerView::type::const_iterator>::difference_type
DistanceType;
@@ -2364,18 +2434,15 @@
typedef internal::StlContainerView<
typename std::remove_const<LhsContainer>::type>
LhsView;
- typedef typename LhsView::type LhsStlContainer;
StlContainerReference lhs_stl_container = LhsView::ConstReference(lhs);
- if (lhs_stl_container == expected_)
- return true;
+ if (lhs_stl_container == expected_) return true;
::std::ostream* const os = listener->stream();
if (os != nullptr) {
// Something is different. Check for extra values first.
bool printed_header = false;
- for (typename LhsStlContainer::const_iterator it =
- lhs_stl_container.begin();
- it != lhs_stl_container.end(); ++it) {
+ for (auto it = lhs_stl_container.begin(); it != lhs_stl_container.end();
+ ++it) {
if (internal::ArrayAwareFind(expected_.begin(), expected_.end(), *it) ==
expected_.end()) {
if (printed_header) {
@@ -2390,11 +2457,10 @@
// Now check for missing values.
bool printed_header2 = false;
- for (typename StlContainer::const_iterator it = expected_.begin();
- it != expected_.end(); ++it) {
- if (internal::ArrayAwareFind(
- lhs_stl_container.begin(), lhs_stl_container.end(), *it) ==
- lhs_stl_container.end()) {
+ for (auto it = expected_.begin(); it != expected_.end(); ++it) {
+ if (internal::ArrayAwareFind(lhs_stl_container.begin(),
+ lhs_stl_container.end(),
+ *it) == lhs_stl_container.end()) {
if (printed_header2) {
*os << ", ";
} else {
@@ -2417,7 +2483,9 @@
// A comparator functor that uses the < operator to compare two values.
struct LessComparator {
template <typename T, typename U>
- bool operator()(const T& lhs, const U& rhs) const { return lhs < rhs; }
+ bool operator()(const T& lhs, const U& rhs) const {
+ return lhs < rhs;
+ }
};
// Implements WhenSortedBy(comparator, container_matcher).
@@ -2436,14 +2504,16 @@
template <typename LhsContainer>
class Impl : public MatcherInterface<LhsContainer> {
public:
- typedef internal::StlContainerView<
- GTEST_REMOVE_REFERENCE_AND_CONST_(LhsContainer)> LhsView;
+ typedef internal::StlContainerView<GTEST_REMOVE_REFERENCE_AND_CONST_(
+ LhsContainer)>
+ LhsView;
typedef typename LhsView::type LhsStlContainer;
typedef typename LhsView::const_reference LhsStlContainerReference;
// Transforms std::pair<const Key, Value> into std::pair<Key, Value>
// so that we can match associative containers.
- typedef typename RemoveConstFromKey<
- typename LhsStlContainer::value_type>::type LhsValue;
+ typedef
+ typename RemoveConstFromKey<typename LhsStlContainer::value_type>::type
+ LhsValue;
Impl(const Comparator& comparator, const ContainerMatcher& matcher)
: comparator_(comparator), matcher_(matcher) {}
@@ -2463,8 +2533,8 @@
LhsStlContainerReference lhs_stl_container = LhsView::ConstReference(lhs);
::std::vector<LhsValue> sorted_container(lhs_stl_container.begin(),
lhs_stl_container.end());
- ::std::sort(
- sorted_container.begin(), sorted_container.end(), comparator_);
+ ::std::sort(sorted_container.begin(), sorted_container.end(),
+ comparator_);
if (!listener->IsInterested()) {
// If the listener is not interested, we do not need to
@@ -2477,8 +2547,8 @@
*listener << " when sorted";
StringMatchResultListener inner_listener;
- const bool match = matcher_.MatchAndExplain(sorted_container,
- &inner_listener);
+ const bool match =
+ matcher_.MatchAndExplain(sorted_container, &inner_listener);
PrintIfNotEmpty(inner_listener.str(), listener->stream());
return match;
}
@@ -2487,7 +2557,8 @@
const Comparator comparator_;
const Matcher<const ::std::vector<LhsValue>&> matcher_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(Impl);
+ Impl(const Impl&) = delete;
+ Impl& operator=(const Impl&) = delete;
};
private:
@@ -2501,9 +2572,9 @@
// container and the RHS container respectively.
template <typename TupleMatcher, typename RhsContainer>
class PointwiseMatcher {
- GTEST_COMPILE_ASSERT_(
+ static_assert(
!IsHashTable<GTEST_REMOVE_REFERENCE_AND_CONST_(RhsContainer)>::value,
- use_UnorderedPointwise_with_hash_tables);
+ "use UnorderedPointwise with hash tables");
public:
typedef internal::StlContainerView<RhsContainer> RhsView;
@@ -2522,9 +2593,9 @@
template <typename LhsContainer>
operator Matcher<LhsContainer>() const {
- GTEST_COMPILE_ASSERT_(
+ static_assert(
!IsHashTable<GTEST_REMOVE_REFERENCE_AND_CONST_(LhsContainer)>::value,
- use_UnorderedPointwise_with_hash_tables);
+ "use UnorderedPointwise with hash tables");
return Matcher<LhsContainer>(
new Impl<const LhsContainer&>(tuple_matcher_, rhs_));
@@ -2533,8 +2604,9 @@
template <typename LhsContainer>
class Impl : public MatcherInterface<LhsContainer> {
public:
- typedef internal::StlContainerView<
- GTEST_REMOVE_REFERENCE_AND_CONST_(LhsContainer)> LhsView;
+ typedef internal::StlContainerView<GTEST_REMOVE_REFERENCE_AND_CONST_(
+ LhsContainer)>
+ LhsView;
typedef typename LhsView::type LhsStlContainer;
typedef typename LhsView::const_reference LhsStlContainerReference;
typedef typename LhsStlContainer::value_type LhsValue;
@@ -2574,14 +2646,14 @@
return false;
}
- typename LhsStlContainer::const_iterator left = lhs_stl_container.begin();
- typename RhsStlContainer::const_iterator right = rhs_.begin();
+ auto left = lhs_stl_container.begin();
+ auto right = rhs_.begin();
for (size_t i = 0; i != actual_size; ++i, ++left, ++right) {
if (listener->IsInterested()) {
StringMatchResultListener inner_listener;
// Create InnerMatcherArg as a temporarily object to avoid it outlives
// *left and *right. Dereference or the conversion to `const T&` may
- // return temp objects, e.g for vector<bool>.
+ // return temp objects, e.g. for vector<bool>.
if (!mono_tuple_matcher_.MatchAndExplain(
InnerMatcherArg(ImplicitCast_<const LhsValue&>(*left),
ImplicitCast_<const RhsValue&>(*right)),
@@ -2628,18 +2700,17 @@
template <typename InnerMatcher>
explicit QuantifierMatcherImpl(InnerMatcher inner_matcher)
: inner_matcher_(
- testing::SafeMatcherCast<const Element&>(inner_matcher)) {}
+ testing::SafeMatcherCast<const Element&>(inner_matcher)) {}
// Checks whether:
// * All elements in the container match, if all_elements_should_match.
// * Any element in the container matches, if !all_elements_should_match.
- bool MatchAndExplainImpl(bool all_elements_should_match,
- Container container,
+ bool MatchAndExplainImpl(bool all_elements_should_match, Container container,
MatchResultListener* listener) const {
StlContainerReference stl_container = View::ConstReference(container);
size_t i = 0;
- for (typename StlContainer::const_iterator it = stl_container.begin();
- it != stl_container.end(); ++it, ++i) {
+ for (auto it = stl_container.begin(); it != stl_container.end();
+ ++it, ++i) {
StringMatchResultListener inner_listener;
const bool matches = inner_matcher_.MatchAndExplain(*it, &inner_listener);
@@ -2653,6 +2724,54 @@
return all_elements_should_match;
}
+ bool MatchAndExplainImpl(const Matcher<size_t>& count_matcher,
+ Container container,
+ MatchResultListener* listener) const {
+ StlContainerReference stl_container = View::ConstReference(container);
+ size_t i = 0;
+ std::vector<size_t> match_elements;
+ for (auto it = stl_container.begin(); it != stl_container.end();
+ ++it, ++i) {
+ StringMatchResultListener inner_listener;
+ const bool matches = inner_matcher_.MatchAndExplain(*it, &inner_listener);
+ if (matches) {
+ match_elements.push_back(i);
+ }
+ }
+ if (listener->IsInterested()) {
+ if (match_elements.empty()) {
+ *listener << "has no element that matches";
+ } else if (match_elements.size() == 1) {
+ *listener << "whose element #" << match_elements[0] << " matches";
+ } else {
+ *listener << "whose elements (";
+ std::string sep = "";
+ for (size_t e : match_elements) {
+ *listener << sep << e;
+ sep = ", ";
+ }
+ *listener << ") match";
+ }
+ }
+ StringMatchResultListener count_listener;
+ if (count_matcher.MatchAndExplain(match_elements.size(), &count_listener)) {
+ *listener << " and whose match quantity of " << match_elements.size()
+ << " matches";
+ PrintIfNotEmpty(count_listener.str(), listener->stream());
+ return true;
+ } else {
+ if (match_elements.empty()) {
+ *listener << " and";
+ } else {
+ *listener << " but";
+ }
+ *listener << " whose match quantity of " << match_elements.size()
+ << " does not match";
+ PrintIfNotEmpty(count_listener.str(), listener->stream());
+ return false;
+ }
+ }
+
protected:
const Matcher<const Element&> inner_matcher_;
};
@@ -2709,6 +2828,58 @@
}
};
+// Implements Contains(element_matcher).Times(n) for the given argument type
+// Container.
+template <typename Container>
+class ContainsTimesMatcherImpl : public QuantifierMatcherImpl<Container> {
+ public:
+ template <typename InnerMatcher>
+ explicit ContainsTimesMatcherImpl(InnerMatcher inner_matcher,
+ Matcher<size_t> count_matcher)
+ : QuantifierMatcherImpl<Container>(inner_matcher),
+ count_matcher_(std::move(count_matcher)) {}
+
+ void DescribeTo(::std::ostream* os) const override {
+ *os << "quantity of elements that match ";
+ this->inner_matcher_.DescribeTo(os);
+ *os << " ";
+ count_matcher_.DescribeTo(os);
+ }
+
+ void DescribeNegationTo(::std::ostream* os) const override {
+ *os << "quantity of elements that match ";
+ this->inner_matcher_.DescribeTo(os);
+ *os << " ";
+ count_matcher_.DescribeNegationTo(os);
+ }
+
+ bool MatchAndExplain(Container container,
+ MatchResultListener* listener) const override {
+ return this->MatchAndExplainImpl(count_matcher_, container, listener);
+ }
+
+ private:
+ const Matcher<size_t> count_matcher_;
+};
+
+// Implements polymorphic Contains(element_matcher).Times(n).
+template <typename M>
+class ContainsTimesMatcher {
+ public:
+ explicit ContainsTimesMatcher(M m, Matcher<size_t> count_matcher)
+ : inner_matcher_(m), count_matcher_(std::move(count_matcher)) {}
+
+ template <typename Container>
+ operator Matcher<Container>() const { // NOLINT
+ return Matcher<Container>(new ContainsTimesMatcherImpl<const Container&>(
+ inner_matcher_, count_matcher_));
+ }
+
+ private:
+ const M inner_matcher_;
+ const Matcher<size_t> count_matcher_;
+};
+
// Implements polymorphic Contains(element_matcher).
template <typename M>
class ContainsMatcher {
@@ -2716,11 +2887,15 @@
explicit ContainsMatcher(M m) : inner_matcher_(m) {}
template <typename Container>
- operator Matcher<Container>() const {
+ operator Matcher<Container>() const { // NOLINT
return Matcher<Container>(
new ContainsMatcherImpl<const Container&>(inner_matcher_));
}
+ ContainsTimesMatcher<M> Times(Matcher<size_t> count_matcher) const {
+ return ContainsTimesMatcher<M>(inner_matcher_, std::move(count_matcher));
+ }
+
private:
const M inner_matcher_;
};
@@ -2732,7 +2907,7 @@
explicit EachMatcher(M m) : inner_matcher_(m) {}
template <typename Container>
- operator Matcher<Container>() const {
+ operator Matcher<Container>() const { // NOLINT
return Matcher<Container>(
new EachMatcherImpl<const Container&>(inner_matcher_));
}
@@ -2778,8 +2953,7 @@
template <typename InnerMatcher>
explicit KeyMatcherImpl(InnerMatcher inner_matcher)
: inner_matcher_(
- testing::SafeMatcherCast<const KeyType&>(inner_matcher)) {
- }
+ testing::SafeMatcherCast<const KeyType&>(inner_matcher)) {}
// Returns true if and only if 'key_value.first' (the key) matches the inner
// matcher.
@@ -2884,8 +3058,7 @@
: first_matcher_(
testing::SafeMatcherCast<const FirstType&>(first_matcher)),
second_matcher_(
- testing::SafeMatcherCast<const SecondType&>(second_matcher)) {
- }
+ testing::SafeMatcherCast<const SecondType&>(second_matcher)) {}
// Describes what this matcher does.
void DescribeTo(::std::ostream* os) const override {
@@ -2963,7 +3136,7 @@
: first_matcher_(first_matcher), second_matcher_(second_matcher) {}
template <typename PairType>
- operator Matcher<PairType> () const {
+ operator Matcher<PairType>() const {
return Matcher<PairType>(
new PairMatcherImpl<const PairType&>(first_matcher_, second_matcher_));
}
@@ -3235,7 +3408,7 @@
// explanations[i] is the explanation of the element at index i.
::std::vector<std::string> explanations(count());
StlContainerReference stl_container = View::ConstReference(container);
- typename StlContainer::const_iterator it = stl_container.begin();
+ auto it = stl_container.begin();
size_t exam_pos = 0;
bool mismatch_found = false; // Have we found a mismatched element yet?
@@ -3312,7 +3485,7 @@
size_t count() const { return matchers_.size(); }
- ::std::vector<Matcher<const Element&> > matchers_;
+ ::std::vector<Matcher<const Element&>> matchers_;
};
// Connectivity matrix of (elements X matchers), in element-major order.
@@ -3324,8 +3497,7 @@
MatchMatrix(size_t num_elements, size_t num_matchers)
: num_elements_(num_elements),
num_matchers_(num_matchers),
- matched_(num_elements_* num_matchers_, 0) {
- }
+ matched_(num_elements_ * num_matchers_, 0) {}
size_t LhsSize() const { return num_elements_; }
size_t RhsSize() const { return num_matchers_; }
@@ -3364,8 +3536,7 @@
// Returns a maximum bipartite matching for the specified graph 'g'.
// The matching is represented as a vector of {element, matcher} pairs.
-GTEST_API_ ElementMatcherPairs
-FindMaxBipartiteMatching(const MatchMatrix& g);
+GTEST_API_ ElementMatcherPairs FindMaxBipartiteMatching(const MatchMatrix& g);
struct UnorderedMatcherRequire {
enum Flags {
@@ -3402,9 +3573,7 @@
bool FindPairing(const MatchMatrix& matrix,
MatchResultListener* listener) const;
- MatcherDescriberVec& matcher_describers() {
- return matcher_describers_;
- }
+ MatcherDescriberVec& matcher_describers() { return matcher_describers_; }
static Message Elements(size_t n) {
return Message() << n << " element" << (n == 1 ? "" : "s");
@@ -3428,7 +3597,6 @@
typedef internal::StlContainerView<RawContainer> View;
typedef typename View::type StlContainer;
typedef typename View::const_reference StlContainerReference;
- typedef typename StlContainer::const_iterator StlContainerConstIterator;
typedef typename StlContainer::value_type Element;
template <typename InputIter>
@@ -3511,7 +3679,7 @@
return matrix;
}
- ::std::vector<Matcher<const Element&> > matchers_;
+ ::std::vector<Matcher<const Element&>> matchers_;
};
// Functor for use in TransformTuple.
@@ -3536,7 +3704,7 @@
typedef GTEST_REMOVE_REFERENCE_AND_CONST_(Container) RawContainer;
typedef typename internal::StlContainerView<RawContainer>::type View;
typedef typename View::value_type Element;
- typedef ::std::vector<Matcher<const Element&> > MatcherVec;
+ typedef ::std::vector<Matcher<const Element&>> MatcherVec;
MatcherVec matchers;
matchers.reserve(::std::tuple_size<MatcherTuple>::value);
TransformTupleValues(CastAndAppendTransform<const Element&>(), matchers_,
@@ -3559,15 +3727,15 @@
template <typename Container>
operator Matcher<Container>() const {
- GTEST_COMPILE_ASSERT_(
+ static_assert(
!IsHashTable<GTEST_REMOVE_REFERENCE_AND_CONST_(Container)>::value ||
::std::tuple_size<MatcherTuple>::value < 2,
- use_UnorderedElementsAre_with_hash_tables);
+ "use UnorderedElementsAre with hash tables");
typedef GTEST_REMOVE_REFERENCE_AND_CONST_(Container) RawContainer;
typedef typename internal::StlContainerView<RawContainer>::type View;
typedef typename View::value_type Element;
- typedef ::std::vector<Matcher<const Element&> > MatcherVec;
+ typedef ::std::vector<Matcher<const Element&>> MatcherVec;
MatcherVec matchers;
matchers.reserve(::std::tuple_size<MatcherTuple>::value);
TransformTupleValues(CastAndAppendTransform<const Element&>(), matchers_,
@@ -3610,9 +3778,9 @@
template <typename Container>
operator Matcher<Container>() const {
- GTEST_COMPILE_ASSERT_(
+ static_assert(
!IsHashTable<GTEST_REMOVE_REFERENCE_AND_CONST_(Container)>::value,
- use_UnorderedElementsAreArray_with_hash_tables);
+ "use UnorderedElementsAreArray with hash tables");
return Matcher<Container>(new ElementsAreMatcherImpl<const Container&>(
matchers_.begin(), matchers_.end()));
@@ -3702,9 +3870,9 @@
// 'negation' is false; otherwise returns the description of the
// negation of the matcher. 'param_values' contains a list of strings
// that are the print-out of the matcher's parameters.
-GTEST_API_ std::string FormatMatcherDescription(bool negation,
- const char* matcher_name,
- const Strings& param_values);
+GTEST_API_ std::string FormatMatcherDescription(
+ bool negation, const char* matcher_name,
+ const std::vector<const char*>& param_names, const Strings& param_values);
// Implements a matcher that checks the value of a optional<> type variable.
template <typename ValueMatcher>
@@ -3981,26 +4149,26 @@
}
template <typename T>
-inline internal::ElementsAreArrayMatcher<T> ElementsAreArray(
- const T* pointer, size_t count) {
+inline auto ElementsAreArray(const T* pointer, size_t count)
+ -> decltype(ElementsAreArray(pointer, pointer + count)) {
return ElementsAreArray(pointer, pointer + count);
}
template <typename T, size_t N>
-inline internal::ElementsAreArrayMatcher<T> ElementsAreArray(
- const T (&array)[N]) {
+inline auto ElementsAreArray(const T (&array)[N])
+ -> decltype(ElementsAreArray(array, N)) {
return ElementsAreArray(array, N);
}
template <typename Container>
-inline internal::ElementsAreArrayMatcher<typename Container::value_type>
-ElementsAreArray(const Container& container) {
+inline auto ElementsAreArray(const Container& container)
+ -> decltype(ElementsAreArray(container.begin(), container.end())) {
return ElementsAreArray(container.begin(), container.end());
}
template <typename T>
-inline internal::ElementsAreArrayMatcher<T>
-ElementsAreArray(::std::initializer_list<T> xs) {
+inline auto ElementsAreArray(::std::initializer_list<T> xs)
+ -> decltype(ElementsAreArray(xs.begin(), xs.end())) {
return ElementsAreArray(xs.begin(), xs.end());
}
@@ -4027,14 +4195,14 @@
}
template <typename T>
-inline internal::UnorderedElementsAreArrayMatcher<T>
-UnorderedElementsAreArray(const T* pointer, size_t count) {
+inline internal::UnorderedElementsAreArrayMatcher<T> UnorderedElementsAreArray(
+ const T* pointer, size_t count) {
return UnorderedElementsAreArray(pointer, pointer + count);
}
template <typename T, size_t N>
-inline internal::UnorderedElementsAreArrayMatcher<T>
-UnorderedElementsAreArray(const T (&array)[N]) {
+inline internal::UnorderedElementsAreArrayMatcher<T> UnorderedElementsAreArray(
+ const T (&array)[N]) {
return UnorderedElementsAreArray(array, N);
}
@@ -4046,8 +4214,8 @@
}
template <typename T>
-inline internal::UnorderedElementsAreArrayMatcher<T>
-UnorderedElementsAreArray(::std::initializer_list<T> xs) {
+inline internal::UnorderedElementsAreArrayMatcher<T> UnorderedElementsAreArray(
+ ::std::initializer_list<T> xs) {
return UnorderedElementsAreArray(xs.begin(), xs.end());
}
@@ -4081,14 +4249,14 @@
}
// Creates a polymorphic matcher that matches any NULL pointer.
-inline PolymorphicMatcher<internal::IsNullMatcher > IsNull() {
+inline PolymorphicMatcher<internal::IsNullMatcher> IsNull() {
return MakePolymorphicMatcher(internal::IsNullMatcher());
}
// Creates a polymorphic matcher that matches any non-NULL pointer.
// This is convenient as Not(NULL) doesn't compile (the compiler
// thinks that that expression is comparing a pointer with an integer).
-inline PolymorphicMatcher<internal::NotNullMatcher > NotNull() {
+inline PolymorphicMatcher<internal::NotNullMatcher> NotNull() {
return MakePolymorphicMatcher(internal::NotNullMatcher());
}
@@ -4119,8 +4287,8 @@
// Creates a matcher that matches any double argument approximately equal to
// rhs, up to the specified max absolute error bound, where two NANs are
// considered unequal. The max absolute error bound must be non-negative.
-inline internal::FloatingEqMatcher<double> DoubleNear(
- double rhs, double max_abs_error) {
+inline internal::FloatingEqMatcher<double> DoubleNear(double rhs,
+ double max_abs_error) {
return internal::FloatingEqMatcher<double>(rhs, false, max_abs_error);
}
@@ -4147,8 +4315,8 @@
// Creates a matcher that matches any float argument approximately equal to
// rhs, up to the specified max absolute error bound, where two NANs are
// considered unequal. The max absolute error bound must be non-negative.
-inline internal::FloatingEqMatcher<float> FloatNear(
- float rhs, float max_abs_error) {
+inline internal::FloatingEqMatcher<float> FloatNear(float rhs,
+ float max_abs_error) {
return internal::FloatingEqMatcher<float>(rhs, false, max_abs_error);
}
@@ -4176,7 +4344,7 @@
// If To is a reference and the cast fails, this matcher returns false
// immediately.
template <typename To>
-inline PolymorphicMatcher<internal::WhenDynamicCastToMatcher<To> >
+inline PolymorphicMatcher<internal::WhenDynamicCastToMatcher<To>>
WhenDynamicCastTo(const Matcher<To>& inner_matcher) {
return MakePolymorphicMatcher(
internal::WhenDynamicCastToMatcher<To>(inner_matcher));
@@ -4188,12 +4356,10 @@
// Field(&Foo::number, Ge(5))
// matches a Foo object x if and only if x.number >= 5.
template <typename Class, typename FieldType, typename FieldMatcher>
-inline PolymorphicMatcher<
- internal::FieldMatcher<Class, FieldType> > Field(
+inline PolymorphicMatcher<internal::FieldMatcher<Class, FieldType>> Field(
FieldType Class::*field, const FieldMatcher& matcher) {
- return MakePolymorphicMatcher(
- internal::FieldMatcher<Class, FieldType>(
- field, MatcherCast<const FieldType&>(matcher)));
+ return MakePolymorphicMatcher(internal::FieldMatcher<Class, FieldType>(
+ field, MatcherCast<const FieldType&>(matcher)));
// The call to MatcherCast() is required for supporting inner
// matchers of compatible types. For example, it allows
// Field(&Foo::bar, m)
@@ -4203,7 +4369,7 @@
// Same as Field() but also takes the name of the field to provide better error
// messages.
template <typename Class, typename FieldType, typename FieldMatcher>
-inline PolymorphicMatcher<internal::FieldMatcher<Class, FieldType> > Field(
+inline PolymorphicMatcher<internal::FieldMatcher<Class, FieldType>> Field(
const std::string& field_name, FieldType Class::*field,
const FieldMatcher& matcher) {
return MakePolymorphicMatcher(internal::FieldMatcher<Class, FieldType>(
@@ -4216,7 +4382,7 @@
// matches a Foo object x if and only if x.str() starts with "hi".
template <typename Class, typename PropertyType, typename PropertyMatcher>
inline PolymorphicMatcher<internal::PropertyMatcher<
- Class, PropertyType, PropertyType (Class::*)() const> >
+ Class, PropertyType, PropertyType (Class::*)() const>>
Property(PropertyType (Class::*property)() const,
const PropertyMatcher& matcher) {
return MakePolymorphicMatcher(
@@ -4233,7 +4399,7 @@
// better error messages.
template <typename Class, typename PropertyType, typename PropertyMatcher>
inline PolymorphicMatcher<internal::PropertyMatcher<
- Class, PropertyType, PropertyType (Class::*)() const> >
+ Class, PropertyType, PropertyType (Class::*)() const>>
Property(const std::string& property_name,
PropertyType (Class::*property)() const,
const PropertyMatcher& matcher) {
@@ -4246,8 +4412,8 @@
// The same as above but for reference-qualified member functions.
template <typename Class, typename PropertyType, typename PropertyMatcher>
inline PolymorphicMatcher<internal::PropertyMatcher<
- Class, PropertyType, PropertyType (Class::*)() const &> >
-Property(PropertyType (Class::*property)() const &,
+ Class, PropertyType, PropertyType (Class::*)() const&>>
+Property(PropertyType (Class::*property)() const&,
const PropertyMatcher& matcher) {
return MakePolymorphicMatcher(
internal::PropertyMatcher<Class, PropertyType,
@@ -4258,9 +4424,9 @@
// Three-argument form for reference-qualified member functions.
template <typename Class, typename PropertyType, typename PropertyMatcher>
inline PolymorphicMatcher<internal::PropertyMatcher<
- Class, PropertyType, PropertyType (Class::*)() const &> >
+ Class, PropertyType, PropertyType (Class::*)() const&>>
Property(const std::string& property_name,
- PropertyType (Class::*property)() const &,
+ PropertyType (Class::*property)() const&,
const PropertyMatcher& matcher) {
return MakePolymorphicMatcher(
internal::PropertyMatcher<Class, PropertyType,
@@ -4279,15 +4445,25 @@
template <typename Callable, typename InnerMatcher>
internal::ResultOfMatcher<Callable, InnerMatcher> ResultOf(
Callable callable, InnerMatcher matcher) {
+ return internal::ResultOfMatcher<Callable, InnerMatcher>(std::move(callable),
+ std::move(matcher));
+}
+
+// Same as ResultOf() above, but also takes a description of the `callable`
+// result to provide better error messages.
+template <typename Callable, typename InnerMatcher>
+internal::ResultOfMatcher<Callable, InnerMatcher> ResultOf(
+ const std::string& result_description, Callable callable,
+ InnerMatcher matcher) {
return internal::ResultOfMatcher<Callable, InnerMatcher>(
- std::move(callable), std::move(matcher));
+ result_description, std::move(callable), std::move(matcher));
}
// String matchers.
// Matches a string equal to str.
template <typename T = std::string>
-PolymorphicMatcher<internal::StrEqualityMatcher<std::string> > StrEq(
+PolymorphicMatcher<internal::StrEqualityMatcher<std::string>> StrEq(
const internal::StringLike<T>& str) {
return MakePolymorphicMatcher(
internal::StrEqualityMatcher<std::string>(std::string(str), true, true));
@@ -4295,7 +4471,7 @@
// Matches a string not equal to str.
template <typename T = std::string>
-PolymorphicMatcher<internal::StrEqualityMatcher<std::string> > StrNe(
+PolymorphicMatcher<internal::StrEqualityMatcher<std::string>> StrNe(
const internal::StringLike<T>& str) {
return MakePolymorphicMatcher(
internal::StrEqualityMatcher<std::string>(std::string(str), false, true));
@@ -4303,7 +4479,7 @@
// Matches a string equal to str, ignoring case.
template <typename T = std::string>
-PolymorphicMatcher<internal::StrEqualityMatcher<std::string> > StrCaseEq(
+PolymorphicMatcher<internal::StrEqualityMatcher<std::string>> StrCaseEq(
const internal::StringLike<T>& str) {
return MakePolymorphicMatcher(
internal::StrEqualityMatcher<std::string>(std::string(str), true, false));
@@ -4311,7 +4487,7 @@
// Matches a string not equal to str, ignoring case.
template <typename T = std::string>
-PolymorphicMatcher<internal::StrEqualityMatcher<std::string> > StrCaseNe(
+PolymorphicMatcher<internal::StrEqualityMatcher<std::string>> StrCaseNe(
const internal::StringLike<T>& str) {
return MakePolymorphicMatcher(internal::StrEqualityMatcher<std::string>(
std::string(str), false, false));
@@ -4320,7 +4496,7 @@
// Creates a matcher that matches any string, std::string, or C string
// that contains the given substring.
template <typename T = std::string>
-PolymorphicMatcher<internal::HasSubstrMatcher<std::string> > HasSubstr(
+PolymorphicMatcher<internal::HasSubstrMatcher<std::string>> HasSubstr(
const internal::StringLike<T>& substring) {
return MakePolymorphicMatcher(
internal::HasSubstrMatcher<std::string>(std::string(substring)));
@@ -4328,7 +4504,7 @@
// Matches a string that starts with 'prefix' (case-sensitive).
template <typename T = std::string>
-PolymorphicMatcher<internal::StartsWithMatcher<std::string> > StartsWith(
+PolymorphicMatcher<internal::StartsWithMatcher<std::string>> StartsWith(
const internal::StringLike<T>& prefix) {
return MakePolymorphicMatcher(
internal::StartsWithMatcher<std::string>(std::string(prefix)));
@@ -4336,7 +4512,7 @@
// Matches a string that ends with 'suffix' (case-sensitive).
template <typename T = std::string>
-PolymorphicMatcher<internal::EndsWithMatcher<std::string> > EndsWith(
+PolymorphicMatcher<internal::EndsWithMatcher<std::string>> EndsWith(
const internal::StringLike<T>& suffix) {
return MakePolymorphicMatcher(
internal::EndsWithMatcher<std::string>(std::string(suffix)));
@@ -4346,50 +4522,50 @@
// Wide string matchers.
// Matches a string equal to str.
-inline PolymorphicMatcher<internal::StrEqualityMatcher<std::wstring> > StrEq(
+inline PolymorphicMatcher<internal::StrEqualityMatcher<std::wstring>> StrEq(
const std::wstring& str) {
return MakePolymorphicMatcher(
internal::StrEqualityMatcher<std::wstring>(str, true, true));
}
// Matches a string not equal to str.
-inline PolymorphicMatcher<internal::StrEqualityMatcher<std::wstring> > StrNe(
+inline PolymorphicMatcher<internal::StrEqualityMatcher<std::wstring>> StrNe(
const std::wstring& str) {
return MakePolymorphicMatcher(
internal::StrEqualityMatcher<std::wstring>(str, false, true));
}
// Matches a string equal to str, ignoring case.
-inline PolymorphicMatcher<internal::StrEqualityMatcher<std::wstring> >
-StrCaseEq(const std::wstring& str) {
+inline PolymorphicMatcher<internal::StrEqualityMatcher<std::wstring>> StrCaseEq(
+ const std::wstring& str) {
return MakePolymorphicMatcher(
internal::StrEqualityMatcher<std::wstring>(str, true, false));
}
// Matches a string not equal to str, ignoring case.
-inline PolymorphicMatcher<internal::StrEqualityMatcher<std::wstring> >
-StrCaseNe(const std::wstring& str) {
+inline PolymorphicMatcher<internal::StrEqualityMatcher<std::wstring>> StrCaseNe(
+ const std::wstring& str) {
return MakePolymorphicMatcher(
internal::StrEqualityMatcher<std::wstring>(str, false, false));
}
// Creates a matcher that matches any ::wstring, std::wstring, or C wide string
// that contains the given substring.
-inline PolymorphicMatcher<internal::HasSubstrMatcher<std::wstring> > HasSubstr(
+inline PolymorphicMatcher<internal::HasSubstrMatcher<std::wstring>> HasSubstr(
const std::wstring& substring) {
return MakePolymorphicMatcher(
internal::HasSubstrMatcher<std::wstring>(substring));
}
// Matches a string that starts with 'prefix' (case-sensitive).
-inline PolymorphicMatcher<internal::StartsWithMatcher<std::wstring> >
-StartsWith(const std::wstring& prefix) {
+inline PolymorphicMatcher<internal::StartsWithMatcher<std::wstring>> StartsWith(
+ const std::wstring& prefix) {
return MakePolymorphicMatcher(
internal::StartsWithMatcher<std::wstring>(prefix));
}
// Matches a string that ends with 'suffix' (case-sensitive).
-inline PolymorphicMatcher<internal::EndsWithMatcher<std::wstring> > EndsWith(
+inline PolymorphicMatcher<internal::EndsWithMatcher<std::wstring>> EndsWith(
const std::wstring& suffix) {
return MakePolymorphicMatcher(
internal::EndsWithMatcher<std::wstring>(suffix));
@@ -4484,8 +4660,8 @@
// predicate. The predicate can be any unary function or functor
// whose return type can be implicitly converted to bool.
template <typename Predicate>
-inline PolymorphicMatcher<internal::TrulyMatcher<Predicate> >
-Truly(Predicate pred) {
+inline PolymorphicMatcher<internal::TrulyMatcher<Predicate>> Truly(
+ Predicate pred) {
return MakePolymorphicMatcher(internal::TrulyMatcher<Predicate>(pred));
}
@@ -4496,8 +4672,8 @@
// EXPECT_THAT(container, SizeIs(2)); // Checks container has 2 elements.
// EXPECT_THAT(container, SizeIs(Le(2)); // Checks container has at most 2.
template <typename SizeMatcher>
-inline internal::SizeIsMatcher<SizeMatcher>
-SizeIs(const SizeMatcher& size_matcher) {
+inline internal::SizeIsMatcher<SizeMatcher> SizeIs(
+ const SizeMatcher& size_matcher) {
return internal::SizeIsMatcher<SizeMatcher>(size_matcher);
}
@@ -4507,8 +4683,8 @@
// do not implement size(). The container must provide const_iterator (with
// valid iterator_traits), begin() and end().
template <typename DistanceMatcher>
-inline internal::BeginEndDistanceIsMatcher<DistanceMatcher>
-BeginEndDistanceIs(const DistanceMatcher& distance_matcher) {
+inline internal::BeginEndDistanceIsMatcher<DistanceMatcher> BeginEndDistanceIs(
+ const DistanceMatcher& distance_matcher) {
return internal::BeginEndDistanceIsMatcher<DistanceMatcher>(distance_matcher);
}
@@ -4517,8 +4693,8 @@
// values that are included in one container but not the other. (Duplicate
// values and order differences are not explained.)
template <typename Container>
-inline PolymorphicMatcher<internal::ContainerEqMatcher<
- typename std::remove_const<Container>::type>>
+inline PolymorphicMatcher<
+ internal::ContainerEqMatcher<typename std::remove_const<Container>::type>>
ContainerEq(const Container& rhs) {
return MakePolymorphicMatcher(internal::ContainerEqMatcher<Container>(rhs));
}
@@ -4526,9 +4702,8 @@
// Returns a matcher that matches a container that, when sorted using
// the given comparator, matches container_matcher.
template <typename Comparator, typename ContainerMatcher>
-inline internal::WhenSortedByMatcher<Comparator, ContainerMatcher>
-WhenSortedBy(const Comparator& comparator,
- const ContainerMatcher& container_matcher) {
+inline internal::WhenSortedByMatcher<Comparator, ContainerMatcher> WhenSortedBy(
+ const Comparator& comparator, const ContainerMatcher& container_matcher) {
return internal::WhenSortedByMatcher<Comparator, ContainerMatcher>(
comparator, container_matcher);
}
@@ -4538,9 +4713,9 @@
template <typename ContainerMatcher>
inline internal::WhenSortedByMatcher<internal::LessComparator, ContainerMatcher>
WhenSorted(const ContainerMatcher& container_matcher) {
- return
- internal::WhenSortedByMatcher<internal::LessComparator, ContainerMatcher>(
- internal::LessComparator(), container_matcher);
+ return internal::WhenSortedByMatcher<internal::LessComparator,
+ ContainerMatcher>(
+ internal::LessComparator(), container_matcher);
}
// Matches an STL-style container or a native array that contains the
@@ -4557,15 +4732,13 @@
rhs);
}
-
// Supports the Pointwise(m, {a, b, c}) syntax.
template <typename TupleMatcher, typename T>
-inline internal::PointwiseMatcher<TupleMatcher, std::vector<T> > Pointwise(
+inline internal::PointwiseMatcher<TupleMatcher, std::vector<T>> Pointwise(
const TupleMatcher& tuple_matcher, std::initializer_list<T> rhs) {
return Pointwise(tuple_matcher, std::vector<T>(rhs));
}
-
// UnorderedPointwise(pair_matcher, rhs) matches an STL-style
// container or a native array that contains the same number of
// elements as in rhs, where in some permutation of the container, its
@@ -4594,28 +4767,25 @@
RhsView::ConstReference(rhs_container);
// Create a matcher for each element in rhs_container.
- ::std::vector<internal::BoundSecondMatcher<Tuple2Matcher, Second> > matchers;
- for (typename RhsStlContainer::const_iterator it = rhs_stl_container.begin();
- it != rhs_stl_container.end(); ++it) {
- matchers.push_back(
- internal::MatcherBindSecond(tuple2_matcher, *it));
+ ::std::vector<internal::BoundSecondMatcher<Tuple2Matcher, Second>> matchers;
+ for (auto it = rhs_stl_container.begin(); it != rhs_stl_container.end();
+ ++it) {
+ matchers.push_back(internal::MatcherBindSecond(tuple2_matcher, *it));
}
// Delegate the work to UnorderedElementsAreArray().
return UnorderedElementsAreArray(matchers);
}
-
// Supports the UnorderedPointwise(m, {a, b, c}) syntax.
template <typename Tuple2Matcher, typename T>
inline internal::UnorderedElementsAreArrayMatcher<
- typename internal::BoundSecondMatcher<Tuple2Matcher, T> >
+ typename internal::BoundSecondMatcher<Tuple2Matcher, T>>
UnorderedPointwise(const Tuple2Matcher& tuple2_matcher,
std::initializer_list<T> rhs) {
return UnorderedPointwise(tuple2_matcher, std::vector<T>(rhs));
}
-
// Matches an STL-style container or a native array that contains at
// least one element matching the given value or matcher.
//
@@ -4625,7 +4795,7 @@
// page_ids.insert(1);
// EXPECT_THAT(page_ids, Contains(1));
// EXPECT_THAT(page_ids, Contains(Gt(2)));
-// EXPECT_THAT(page_ids, Not(Contains(4)));
+// EXPECT_THAT(page_ids, Not(Contains(4))); // See below for Times(0)
//
// ::std::map<int, size_t> page_lengths;
// page_lengths[1] = 100;
@@ -4634,6 +4804,19 @@
//
// const char* user_ids[] = { "joe", "mike", "tom" };
// EXPECT_THAT(user_ids, Contains(Eq(::std::string("tom"))));
+//
+// The matcher supports a modifier `Times` that allows to check for arbitrary
+// occurrences including testing for absence with Times(0).
+//
+// Examples:
+// ::std::vector<int> ids;
+// ids.insert(1);
+// ids.insert(1);
+// ids.insert(3);
+// EXPECT_THAT(ids, Contains(1).Times(2)); // 1 occurs 2 times
+// EXPECT_THAT(ids, Contains(2).Times(0)); // 2 is not present
+// EXPECT_THAT(ids, Contains(3).Times(Ge(1))); // 3 occurs at least once
+
template <typename M>
inline internal::ContainsMatcher<M> Contains(M matcher) {
return internal::ContainsMatcher<M>(matcher);
@@ -4760,7 +4943,7 @@
// Matches an STL-style container or a native array that contains only
// elements matching the given value or matcher.
//
-// Each(m) is semantically equivalent to Not(Contains(Not(m))). Only
+// Each(m) is semantically equivalent to `Not(Contains(Not(m)))`. Only
// the messages are different.
//
// Examples:
@@ -4803,13 +4986,25 @@
// to match a std::map<int, string> that contains exactly one element whose key
// is >= 5 and whose value equals "foo".
template <typename FirstMatcher, typename SecondMatcher>
-inline internal::PairMatcher<FirstMatcher, SecondMatcher>
-Pair(FirstMatcher first_matcher, SecondMatcher second_matcher) {
- return internal::PairMatcher<FirstMatcher, SecondMatcher>(
- first_matcher, second_matcher);
+inline internal::PairMatcher<FirstMatcher, SecondMatcher> Pair(
+ FirstMatcher first_matcher, SecondMatcher second_matcher) {
+ return internal::PairMatcher<FirstMatcher, SecondMatcher>(first_matcher,
+ second_matcher);
}
namespace no_adl {
+// Conditional() creates a matcher that conditionally uses either the first or
+// second matcher provided. For example, we could create an `equal if, and only
+// if' matcher using the Conditional wrapper as follows:
+//
+// EXPECT_THAT(result, Conditional(condition, Eq(expected), Ne(expected)));
+template <typename MatcherTrue, typename MatcherFalse>
+internal::ConditionalMatcher<MatcherTrue, MatcherFalse> Conditional(
+ bool condition, MatcherTrue matcher_true, MatcherFalse matcher_false) {
+ return internal::ConditionalMatcher<MatcherTrue, MatcherFalse>(
+ condition, std::move(matcher_true), std::move(matcher_false));
+}
+
// FieldsAre(matchers...) matches piecewise the fields of compatible structs.
// These include those that support `get<I>(obj)`, and when structured bindings
// are enabled any class that supports them.
@@ -4836,6 +5031,14 @@
const InnerMatcher& inner_matcher) {
return internal::AddressMatcher<InnerMatcher>(inner_matcher);
}
+
+// Matches a base64 escaped string, when the unescaped string matches the
+// internal matcher.
+template <typename MatcherType>
+internal::WhenBase64UnescapedMatcher WhenBase64Unescaped(
+ const MatcherType& internal_matcher) {
+ return internal::WhenBase64UnescapedMatcher(internal_matcher);
+}
} // namespace no_adl
// Returns a predicate that is satisfied by anything that matches the
@@ -4854,8 +5057,8 @@
// Matches the value against the given matcher and explains the match
// result to listener.
template <typename T, typename M>
-inline bool ExplainMatchResult(
- M matcher, const T& value, MatchResultListener* listener) {
+inline bool ExplainMatchResult(M matcher, const T& value,
+ MatchResultListener* listener) {
return SafeMatcherCast<const T&>(matcher).MatchAndExplain(value, listener);
}
@@ -4865,7 +5068,8 @@
//
// MATCHER_P(XAndYThat, matcher,
// "X that " + DescribeMatcher<int>(matcher, negation) +
-// " and Y that " + DescribeMatcher<double>(matcher, negation)) {
+// (negation ? " or" : " and") + " Y that " +
+// DescribeMatcher<double>(matcher, negation)) {
// return ExplainMatchResult(matcher, arg.x(), result_listener) &&
// ExplainMatchResult(matcher, arg.y(), result_listener);
// }
@@ -5014,7 +5218,9 @@
//
// EXPECT_CALL(foo, Bar(_, _)).With(Eq());
template <typename InnerMatcher>
-inline InnerMatcher AllArgs(const InnerMatcher& matcher) { return matcher; }
+inline InnerMatcher AllArgs(const InnerMatcher& matcher) {
+ return matcher;
+}
// Returns a matcher that matches the value of an optional<> type variable.
// The matcher implementation only uses '!arg' and requires that the optional<>
@@ -5032,7 +5238,7 @@
// Returns a matcher that matches the value of a absl::any type variable.
template <typename T>
-PolymorphicMatcher<internal::any_cast_matcher::AnyCastMatcher<T> > AnyWith(
+PolymorphicMatcher<internal::any_cast_matcher::AnyCastMatcher<T>> AnyWith(
const Matcher<const T&>& matcher) {
return MakePolymorphicMatcher(
internal::any_cast_matcher::AnyCastMatcher<T>(matcher));
@@ -5043,7 +5249,7 @@
// functions.
// It is compatible with std::variant.
template <typename T>
-PolymorphicMatcher<internal::variant_matcher::VariantMatcher<T> > VariantWith(
+PolymorphicMatcher<internal::variant_matcher::VariantMatcher<T>> VariantWith(
const Matcher<const T&>& matcher) {
return MakePolymorphicMatcher(
internal::variant_matcher::VariantMatcher<T>(matcher));
@@ -5072,7 +5278,8 @@
template <typename Err>
bool MatchAndExplain(const Err& err, MatchResultListener* listener) const {
- *listener << "which contains .what() that ";
+ *listener << "which contains .what() (of value = " << err.what()
+ << ") that ";
return matcher_.MatchAndExplain(err.what(), listener);
}
@@ -5222,12 +5429,14 @@
// tests. ASSERT_THAT(value, matcher) and EXPECT_THAT(value, matcher)
// succeed if and only if the value matches the matcher. If the assertion
// fails, the value and the description of the matcher will be printed.
-#define ASSERT_THAT(value, matcher) ASSERT_PRED_FORMAT1(\
- ::testing::internal::MakePredicateFormatterFromMatcher(matcher), value)
-#define EXPECT_THAT(value, matcher) EXPECT_PRED_FORMAT1(\
- ::testing::internal::MakePredicateFormatterFromMatcher(matcher), value)
+#define ASSERT_THAT(value, matcher) \
+ ASSERT_PRED_FORMAT1( \
+ ::testing::internal::MakePredicateFormatterFromMatcher(matcher), value)
+#define EXPECT_THAT(value, matcher) \
+ EXPECT_PRED_FORMAT1( \
+ ::testing::internal::MakePredicateFormatterFromMatcher(matcher), value)
-// MATCHER* macroses itself are listed below.
+// MATCHER* macros itself are listed below.
#define MATCHER(name, description) \
class name##Matcher \
: public ::testing::internal::MatcherBaseImpl<name##Matcher> { \
@@ -5248,12 +5457,13 @@
\
private: \
::std::string FormatDescription(bool negation) const { \
+ /* NOLINTNEXTLINE readability-redundant-string-init */ \
::std::string gmock_description = (description); \
if (!gmock_description.empty()) { \
return gmock_description; \
} \
return ::testing::internal::FormatMatcherDescription(negation, #name, \
- {}); \
+ {}, {}); \
} \
}; \
}; \
@@ -5265,33 +5475,41 @@
const
#define MATCHER_P(name, p0, description) \
- GMOCK_INTERNAL_MATCHER(name, name##MatcherP, description, (p0))
-#define MATCHER_P2(name, p0, p1, description) \
- GMOCK_INTERNAL_MATCHER(name, name##MatcherP2, description, (p0, p1))
-#define MATCHER_P3(name, p0, p1, p2, description) \
- GMOCK_INTERNAL_MATCHER(name, name##MatcherP3, description, (p0, p1, p2))
-#define MATCHER_P4(name, p0, p1, p2, p3, description) \
- GMOCK_INTERNAL_MATCHER(name, name##MatcherP4, description, (p0, p1, p2, p3))
+ GMOCK_INTERNAL_MATCHER(name, name##MatcherP, description, (#p0), (p0))
+#define MATCHER_P2(name, p0, p1, description) \
+ GMOCK_INTERNAL_MATCHER(name, name##MatcherP2, description, (#p0, #p1), \
+ (p0, p1))
+#define MATCHER_P3(name, p0, p1, p2, description) \
+ GMOCK_INTERNAL_MATCHER(name, name##MatcherP3, description, (#p0, #p1, #p2), \
+ (p0, p1, p2))
+#define MATCHER_P4(name, p0, p1, p2, p3, description) \
+ GMOCK_INTERNAL_MATCHER(name, name##MatcherP4, description, \
+ (#p0, #p1, #p2, #p3), (p0, p1, p2, p3))
#define MATCHER_P5(name, p0, p1, p2, p3, p4, description) \
GMOCK_INTERNAL_MATCHER(name, name##MatcherP5, description, \
- (p0, p1, p2, p3, p4))
+ (#p0, #p1, #p2, #p3, #p4), (p0, p1, p2, p3, p4))
#define MATCHER_P6(name, p0, p1, p2, p3, p4, p5, description) \
GMOCK_INTERNAL_MATCHER(name, name##MatcherP6, description, \
+ (#p0, #p1, #p2, #p3, #p4, #p5), \
(p0, p1, p2, p3, p4, p5))
#define MATCHER_P7(name, p0, p1, p2, p3, p4, p5, p6, description) \
GMOCK_INTERNAL_MATCHER(name, name##MatcherP7, description, \
+ (#p0, #p1, #p2, #p3, #p4, #p5, #p6), \
(p0, p1, p2, p3, p4, p5, p6))
#define MATCHER_P8(name, p0, p1, p2, p3, p4, p5, p6, p7, description) \
GMOCK_INTERNAL_MATCHER(name, name##MatcherP8, description, \
+ (#p0, #p1, #p2, #p3, #p4, #p5, #p6, #p7), \
(p0, p1, p2, p3, p4, p5, p6, p7))
#define MATCHER_P9(name, p0, p1, p2, p3, p4, p5, p6, p7, p8, description) \
GMOCK_INTERNAL_MATCHER(name, name##MatcherP9, description, \
+ (#p0, #p1, #p2, #p3, #p4, #p5, #p6, #p7, #p8), \
(p0, p1, p2, p3, p4, p5, p6, p7, p8))
#define MATCHER_P10(name, p0, p1, p2, p3, p4, p5, p6, p7, p8, p9, description) \
GMOCK_INTERNAL_MATCHER(name, name##MatcherP10, description, \
+ (#p0, #p1, #p2, #p3, #p4, #p5, #p6, #p7, #p8, #p9), \
(p0, p1, p2, p3, p4, p5, p6, p7, p8, p9))
-#define GMOCK_INTERNAL_MATCHER(name, full_name, description, args) \
+#define GMOCK_INTERNAL_MATCHER(name, full_name, description, arg_names, args) \
template <GMOCK_INTERNAL_MATCHER_TEMPLATE_PARAMS(args)> \
class full_name : public ::testing::internal::MatcherBaseImpl< \
full_name<GMOCK_INTERNAL_MATCHER_TYPE_PARAMS(args)>> { \
@@ -5320,7 +5538,7 @@
return gmock_description; \
} \
return ::testing::internal::FormatMatcherDescription( \
- negation, #name, \
+ negation, #name, {GMOCK_PP_REMOVE_PARENS(arg_names)}, \
::testing::internal::UniversalTersePrintTupleFieldsToStrings( \
::std::tuple<GMOCK_INTERNAL_MATCHER_TYPE_PARAMS(args)>( \
GMOCK_INTERNAL_MATCHER_MEMBERS_USAGE(args)))); \
diff --git a/third_party/googletest/src/googlemock/include/gmock/gmock-more-actions.h b/third_party/googletest/src/googlemock/include/gmock/gmock-more-actions.h
index fd29335..148ac01 100644
--- a/third_party/googletest/src/googlemock/include/gmock/gmock-more-actions.h
+++ b/third_party/googletest/src/googlemock/include/gmock/gmock-more-actions.h
@@ -27,12 +27,12 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Google Mock - a framework for writing C++ mock classes.
//
// This file implements some commonly used variadic actions.
-// GOOGLETEST_CM0002 DO NOT DELETE
+// IWYU pragma: private, include "gmock/gmock.h"
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_MORE_ACTIONS_H_
#define GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_MORE_ACTIONS_H_
@@ -129,170 +129,207 @@
// Declares the template parameters.
#define GMOCK_INTERNAL_DECL_HAS_1_TEMPLATE_PARAMS(kind0, name0) kind0 name0
-#define GMOCK_INTERNAL_DECL_HAS_2_TEMPLATE_PARAMS(kind0, name0, kind1, \
- name1) kind0 name0, kind1 name1
+#define GMOCK_INTERNAL_DECL_HAS_2_TEMPLATE_PARAMS(kind0, name0, kind1, name1) \
+ kind0 name0, kind1 name1
#define GMOCK_INTERNAL_DECL_HAS_3_TEMPLATE_PARAMS(kind0, name0, kind1, name1, \
- kind2, name2) kind0 name0, kind1 name1, kind2 name2
+ kind2, name2) \
+ kind0 name0, kind1 name1, kind2 name2
#define GMOCK_INTERNAL_DECL_HAS_4_TEMPLATE_PARAMS(kind0, name0, kind1, name1, \
- kind2, name2, kind3, name3) kind0 name0, kind1 name1, kind2 name2, \
- kind3 name3
-#define GMOCK_INTERNAL_DECL_HAS_5_TEMPLATE_PARAMS(kind0, name0, kind1, name1, \
- kind2, name2, kind3, name3, kind4, name4) kind0 name0, kind1 name1, \
- kind2 name2, kind3 name3, kind4 name4
+ kind2, name2, kind3, name3) \
+ kind0 name0, kind1 name1, kind2 name2, kind3 name3
+#define GMOCK_INTERNAL_DECL_HAS_5_TEMPLATE_PARAMS( \
+ kind0, name0, kind1, name1, kind2, name2, kind3, name3, kind4, name4) \
+ kind0 name0, kind1 name1, kind2 name2, kind3 name3, kind4 name4
#define GMOCK_INTERNAL_DECL_HAS_6_TEMPLATE_PARAMS(kind0, name0, kind1, name1, \
- kind2, name2, kind3, name3, kind4, name4, kind5, name5) kind0 name0, \
- kind1 name1, kind2 name2, kind3 name3, kind4 name4, kind5 name5
-#define GMOCK_INTERNAL_DECL_HAS_7_TEMPLATE_PARAMS(kind0, name0, kind1, name1, \
- kind2, name2, kind3, name3, kind4, name4, kind5, name5, kind6, \
- name6) kind0 name0, kind1 name1, kind2 name2, kind3 name3, kind4 name4, \
- kind5 name5, kind6 name6
-#define GMOCK_INTERNAL_DECL_HAS_8_TEMPLATE_PARAMS(kind0, name0, kind1, name1, \
- kind2, name2, kind3, name3, kind4, name4, kind5, name5, kind6, name6, \
- kind7, name7) kind0 name0, kind1 name1, kind2 name2, kind3 name3, \
- kind4 name4, kind5 name5, kind6 name6, kind7 name7
-#define GMOCK_INTERNAL_DECL_HAS_9_TEMPLATE_PARAMS(kind0, name0, kind1, name1, \
- kind2, name2, kind3, name3, kind4, name4, kind5, name5, kind6, name6, \
- kind7, name7, kind8, name8) kind0 name0, kind1 name1, kind2 name2, \
- kind3 name3, kind4 name4, kind5 name5, kind6 name6, kind7 name7, \
- kind8 name8
-#define GMOCK_INTERNAL_DECL_HAS_10_TEMPLATE_PARAMS(kind0, name0, kind1, \
- name1, kind2, name2, kind3, name3, kind4, name4, kind5, name5, kind6, \
- name6, kind7, name7, kind8, name8, kind9, name9) kind0 name0, \
- kind1 name1, kind2 name2, kind3 name3, kind4 name4, kind5 name5, \
- kind6 name6, kind7 name7, kind8 name8, kind9 name9
+ kind2, name2, kind3, name3, \
+ kind4, name4, kind5, name5) \
+ kind0 name0, kind1 name1, kind2 name2, kind3 name3, kind4 name4, kind5 name5
+#define GMOCK_INTERNAL_DECL_HAS_7_TEMPLATE_PARAMS( \
+ kind0, name0, kind1, name1, kind2, name2, kind3, name3, kind4, name4, \
+ kind5, name5, kind6, name6) \
+ kind0 name0, kind1 name1, kind2 name2, kind3 name3, kind4 name4, \
+ kind5 name5, kind6 name6
+#define GMOCK_INTERNAL_DECL_HAS_8_TEMPLATE_PARAMS( \
+ kind0, name0, kind1, name1, kind2, name2, kind3, name3, kind4, name4, \
+ kind5, name5, kind6, name6, kind7, name7) \
+ kind0 name0, kind1 name1, kind2 name2, kind3 name3, kind4 name4, \
+ kind5 name5, kind6 name6, kind7 name7
+#define GMOCK_INTERNAL_DECL_HAS_9_TEMPLATE_PARAMS( \
+ kind0, name0, kind1, name1, kind2, name2, kind3, name3, kind4, name4, \
+ kind5, name5, kind6, name6, kind7, name7, kind8, name8) \
+ kind0 name0, kind1 name1, kind2 name2, kind3 name3, kind4 name4, \
+ kind5 name5, kind6 name6, kind7 name7, kind8 name8
+#define GMOCK_INTERNAL_DECL_HAS_10_TEMPLATE_PARAMS( \
+ kind0, name0, kind1, name1, kind2, name2, kind3, name3, kind4, name4, \
+ kind5, name5, kind6, name6, kind7, name7, kind8, name8, kind9, name9) \
+ kind0 name0, kind1 name1, kind2 name2, kind3 name3, kind4 name4, \
+ kind5 name5, kind6 name6, kind7 name7, kind8 name8, kind9 name9
// Lists the template parameters.
#define GMOCK_INTERNAL_LIST_HAS_1_TEMPLATE_PARAMS(kind0, name0) name0
-#define GMOCK_INTERNAL_LIST_HAS_2_TEMPLATE_PARAMS(kind0, name0, kind1, \
- name1) name0, name1
+#define GMOCK_INTERNAL_LIST_HAS_2_TEMPLATE_PARAMS(kind0, name0, kind1, name1) \
+ name0, name1
#define GMOCK_INTERNAL_LIST_HAS_3_TEMPLATE_PARAMS(kind0, name0, kind1, name1, \
- kind2, name2) name0, name1, name2
+ kind2, name2) \
+ name0, name1, name2
#define GMOCK_INTERNAL_LIST_HAS_4_TEMPLATE_PARAMS(kind0, name0, kind1, name1, \
- kind2, name2, kind3, name3) name0, name1, name2, name3
-#define GMOCK_INTERNAL_LIST_HAS_5_TEMPLATE_PARAMS(kind0, name0, kind1, name1, \
- kind2, name2, kind3, name3, kind4, name4) name0, name1, name2, name3, \
- name4
+ kind2, name2, kind3, name3) \
+ name0, name1, name2, name3
+#define GMOCK_INTERNAL_LIST_HAS_5_TEMPLATE_PARAMS( \
+ kind0, name0, kind1, name1, kind2, name2, kind3, name3, kind4, name4) \
+ name0, name1, name2, name3, name4
#define GMOCK_INTERNAL_LIST_HAS_6_TEMPLATE_PARAMS(kind0, name0, kind1, name1, \
- kind2, name2, kind3, name3, kind4, name4, kind5, name5) name0, name1, \
- name2, name3, name4, name5
-#define GMOCK_INTERNAL_LIST_HAS_7_TEMPLATE_PARAMS(kind0, name0, kind1, name1, \
- kind2, name2, kind3, name3, kind4, name4, kind5, name5, kind6, \
- name6) name0, name1, name2, name3, name4, name5, name6
-#define GMOCK_INTERNAL_LIST_HAS_8_TEMPLATE_PARAMS(kind0, name0, kind1, name1, \
- kind2, name2, kind3, name3, kind4, name4, kind5, name5, kind6, name6, \
- kind7, name7) name0, name1, name2, name3, name4, name5, name6, name7
-#define GMOCK_INTERNAL_LIST_HAS_9_TEMPLATE_PARAMS(kind0, name0, kind1, name1, \
- kind2, name2, kind3, name3, kind4, name4, kind5, name5, kind6, name6, \
- kind7, name7, kind8, name8) name0, name1, name2, name3, name4, name5, \
- name6, name7, name8
-#define GMOCK_INTERNAL_LIST_HAS_10_TEMPLATE_PARAMS(kind0, name0, kind1, \
- name1, kind2, name2, kind3, name3, kind4, name4, kind5, name5, kind6, \
- name6, kind7, name7, kind8, name8, kind9, name9) name0, name1, name2, \
- name3, name4, name5, name6, name7, name8, name9
+ kind2, name2, kind3, name3, \
+ kind4, name4, kind5, name5) \
+ name0, name1, name2, name3, name4, name5
+#define GMOCK_INTERNAL_LIST_HAS_7_TEMPLATE_PARAMS( \
+ kind0, name0, kind1, name1, kind2, name2, kind3, name3, kind4, name4, \
+ kind5, name5, kind6, name6) \
+ name0, name1, name2, name3, name4, name5, name6
+#define GMOCK_INTERNAL_LIST_HAS_8_TEMPLATE_PARAMS( \
+ kind0, name0, kind1, name1, kind2, name2, kind3, name3, kind4, name4, \
+ kind5, name5, kind6, name6, kind7, name7) \
+ name0, name1, name2, name3, name4, name5, name6, name7
+#define GMOCK_INTERNAL_LIST_HAS_9_TEMPLATE_PARAMS( \
+ kind0, name0, kind1, name1, kind2, name2, kind3, name3, kind4, name4, \
+ kind5, name5, kind6, name6, kind7, name7, kind8, name8) \
+ name0, name1, name2, name3, name4, name5, name6, name7, name8
+#define GMOCK_INTERNAL_LIST_HAS_10_TEMPLATE_PARAMS( \
+ kind0, name0, kind1, name1, kind2, name2, kind3, name3, kind4, name4, \
+ kind5, name5, kind6, name6, kind7, name7, kind8, name8, kind9, name9) \
+ name0, name1, name2, name3, name4, name5, name6, name7, name8, name9
// Declares the types of value parameters.
#define GMOCK_INTERNAL_DECL_TYPE_AND_0_VALUE_PARAMS()
#define GMOCK_INTERNAL_DECL_TYPE_AND_1_VALUE_PARAMS(p0) , typename p0##_type
-#define GMOCK_INTERNAL_DECL_TYPE_AND_2_VALUE_PARAMS(p0, p1) , \
- typename p0##_type, typename p1##_type
-#define GMOCK_INTERNAL_DECL_TYPE_AND_3_VALUE_PARAMS(p0, p1, p2) , \
- typename p0##_type, typename p1##_type, typename p2##_type
-#define GMOCK_INTERNAL_DECL_TYPE_AND_4_VALUE_PARAMS(p0, p1, p2, p3) , \
- typename p0##_type, typename p1##_type, typename p2##_type, \
- typename p3##_type
-#define GMOCK_INTERNAL_DECL_TYPE_AND_5_VALUE_PARAMS(p0, p1, p2, p3, p4) , \
- typename p0##_type, typename p1##_type, typename p2##_type, \
- typename p3##_type, typename p4##_type
-#define GMOCK_INTERNAL_DECL_TYPE_AND_6_VALUE_PARAMS(p0, p1, p2, p3, p4, p5) , \
- typename p0##_type, typename p1##_type, typename p2##_type, \
- typename p3##_type, typename p4##_type, typename p5##_type
+#define GMOCK_INTERNAL_DECL_TYPE_AND_2_VALUE_PARAMS(p0, p1) \
+ , typename p0##_type, typename p1##_type
+#define GMOCK_INTERNAL_DECL_TYPE_AND_3_VALUE_PARAMS(p0, p1, p2) \
+ , typename p0##_type, typename p1##_type, typename p2##_type
+#define GMOCK_INTERNAL_DECL_TYPE_AND_4_VALUE_PARAMS(p0, p1, p2, p3) \
+ , typename p0##_type, typename p1##_type, typename p2##_type, \
+ typename p3##_type
+#define GMOCK_INTERNAL_DECL_TYPE_AND_5_VALUE_PARAMS(p0, p1, p2, p3, p4) \
+ , typename p0##_type, typename p1##_type, typename p2##_type, \
+ typename p3##_type, typename p4##_type
+#define GMOCK_INTERNAL_DECL_TYPE_AND_6_VALUE_PARAMS(p0, p1, p2, p3, p4, p5) \
+ , typename p0##_type, typename p1##_type, typename p2##_type, \
+ typename p3##_type, typename p4##_type, typename p5##_type
#define GMOCK_INTERNAL_DECL_TYPE_AND_7_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, \
- p6) , typename p0##_type, typename p1##_type, typename p2##_type, \
- typename p3##_type, typename p4##_type, typename p5##_type, \
- typename p6##_type
+ p6) \
+ , typename p0##_type, typename p1##_type, typename p2##_type, \
+ typename p3##_type, typename p4##_type, typename p5##_type, \
+ typename p6##_type
#define GMOCK_INTERNAL_DECL_TYPE_AND_8_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, \
- p6, p7) , typename p0##_type, typename p1##_type, typename p2##_type, \
- typename p3##_type, typename p4##_type, typename p5##_type, \
- typename p6##_type, typename p7##_type
+ p6, p7) \
+ , typename p0##_type, typename p1##_type, typename p2##_type, \
+ typename p3##_type, typename p4##_type, typename p5##_type, \
+ typename p6##_type, typename p7##_type
#define GMOCK_INTERNAL_DECL_TYPE_AND_9_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, \
- p6, p7, p8) , typename p0##_type, typename p1##_type, typename p2##_type, \
- typename p3##_type, typename p4##_type, typename p5##_type, \
- typename p6##_type, typename p7##_type, typename p8##_type
+ p6, p7, p8) \
+ , typename p0##_type, typename p1##_type, typename p2##_type, \
+ typename p3##_type, typename p4##_type, typename p5##_type, \
+ typename p6##_type, typename p7##_type, typename p8##_type
#define GMOCK_INTERNAL_DECL_TYPE_AND_10_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, \
- p6, p7, p8, p9) , typename p0##_type, typename p1##_type, \
- typename p2##_type, typename p3##_type, typename p4##_type, \
- typename p5##_type, typename p6##_type, typename p7##_type, \
- typename p8##_type, typename p9##_type
+ p6, p7, p8, p9) \
+ , typename p0##_type, typename p1##_type, typename p2##_type, \
+ typename p3##_type, typename p4##_type, typename p5##_type, \
+ typename p6##_type, typename p7##_type, typename p8##_type, \
+ typename p9##_type
// Initializes the value parameters.
-#define GMOCK_INTERNAL_INIT_AND_0_VALUE_PARAMS()\
- ()
-#define GMOCK_INTERNAL_INIT_AND_1_VALUE_PARAMS(p0)\
- (p0##_type gmock_p0) : p0(::std::move(gmock_p0))
-#define GMOCK_INTERNAL_INIT_AND_2_VALUE_PARAMS(p0, p1)\
- (p0##_type gmock_p0, p1##_type gmock_p1) : p0(::std::move(gmock_p0)), \
- p1(::std::move(gmock_p1))
-#define GMOCK_INTERNAL_INIT_AND_3_VALUE_PARAMS(p0, p1, p2)\
- (p0##_type gmock_p0, p1##_type gmock_p1, \
- p2##_type gmock_p2) : p0(::std::move(gmock_p0)), \
- p1(::std::move(gmock_p1)), p2(::std::move(gmock_p2))
-#define GMOCK_INTERNAL_INIT_AND_4_VALUE_PARAMS(p0, p1, p2, p3)\
- (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2, \
- p3##_type gmock_p3) : p0(::std::move(gmock_p0)), \
- p1(::std::move(gmock_p1)), p2(::std::move(gmock_p2)), \
+#define GMOCK_INTERNAL_INIT_AND_0_VALUE_PARAMS() ()
+#define GMOCK_INTERNAL_INIT_AND_1_VALUE_PARAMS(p0) \
+ (p0##_type gmock_p0) : p0(::std::move(gmock_p0))
+#define GMOCK_INTERNAL_INIT_AND_2_VALUE_PARAMS(p0, p1) \
+ (p0##_type gmock_p0, p1##_type gmock_p1) \
+ : p0(::std::move(gmock_p0)), p1(::std::move(gmock_p1))
+#define GMOCK_INTERNAL_INIT_AND_3_VALUE_PARAMS(p0, p1, p2) \
+ (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2) \
+ : p0(::std::move(gmock_p0)), \
+ p1(::std::move(gmock_p1)), \
+ p2(::std::move(gmock_p2))
+#define GMOCK_INTERNAL_INIT_AND_4_VALUE_PARAMS(p0, p1, p2, p3) \
+ (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2, \
+ p3##_type gmock_p3) \
+ : p0(::std::move(gmock_p0)), \
+ p1(::std::move(gmock_p1)), \
+ p2(::std::move(gmock_p2)), \
p3(::std::move(gmock_p3))
-#define GMOCK_INTERNAL_INIT_AND_5_VALUE_PARAMS(p0, p1, p2, p3, p4)\
- (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2, \
- p3##_type gmock_p3, p4##_type gmock_p4) : p0(::std::move(gmock_p0)), \
- p1(::std::move(gmock_p1)), p2(::std::move(gmock_p2)), \
- p3(::std::move(gmock_p3)), p4(::std::move(gmock_p4))
-#define GMOCK_INTERNAL_INIT_AND_6_VALUE_PARAMS(p0, p1, p2, p3, p4, p5)\
- (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2, \
- p3##_type gmock_p3, p4##_type gmock_p4, \
- p5##_type gmock_p5) : p0(::std::move(gmock_p0)), \
- p1(::std::move(gmock_p1)), p2(::std::move(gmock_p2)), \
- p3(::std::move(gmock_p3)), p4(::std::move(gmock_p4)), \
+#define GMOCK_INTERNAL_INIT_AND_5_VALUE_PARAMS(p0, p1, p2, p3, p4) \
+ (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2, \
+ p3##_type gmock_p3, p4##_type gmock_p4) \
+ : p0(::std::move(gmock_p0)), \
+ p1(::std::move(gmock_p1)), \
+ p2(::std::move(gmock_p2)), \
+ p3(::std::move(gmock_p3)), \
+ p4(::std::move(gmock_p4))
+#define GMOCK_INTERNAL_INIT_AND_6_VALUE_PARAMS(p0, p1, p2, p3, p4, p5) \
+ (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2, \
+ p3##_type gmock_p3, p4##_type gmock_p4, p5##_type gmock_p5) \
+ : p0(::std::move(gmock_p0)), \
+ p1(::std::move(gmock_p1)), \
+ p2(::std::move(gmock_p2)), \
+ p3(::std::move(gmock_p3)), \
+ p4(::std::move(gmock_p4)), \
p5(::std::move(gmock_p5))
-#define GMOCK_INTERNAL_INIT_AND_7_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6)\
- (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2, \
- p3##_type gmock_p3, p4##_type gmock_p4, p5##_type gmock_p5, \
- p6##_type gmock_p6) : p0(::std::move(gmock_p0)), \
- p1(::std::move(gmock_p1)), p2(::std::move(gmock_p2)), \
- p3(::std::move(gmock_p3)), p4(::std::move(gmock_p4)), \
- p5(::std::move(gmock_p5)), p6(::std::move(gmock_p6))
-#define GMOCK_INTERNAL_INIT_AND_8_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, p7)\
- (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2, \
- p3##_type gmock_p3, p4##_type gmock_p4, p5##_type gmock_p5, \
- p6##_type gmock_p6, p7##_type gmock_p7) : p0(::std::move(gmock_p0)), \
- p1(::std::move(gmock_p1)), p2(::std::move(gmock_p2)), \
- p3(::std::move(gmock_p3)), p4(::std::move(gmock_p4)), \
- p5(::std::move(gmock_p5)), p6(::std::move(gmock_p6)), \
+#define GMOCK_INTERNAL_INIT_AND_7_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6) \
+ (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2, \
+ p3##_type gmock_p3, p4##_type gmock_p4, p5##_type gmock_p5, \
+ p6##_type gmock_p6) \
+ : p0(::std::move(gmock_p0)), \
+ p1(::std::move(gmock_p1)), \
+ p2(::std::move(gmock_p2)), \
+ p3(::std::move(gmock_p3)), \
+ p4(::std::move(gmock_p4)), \
+ p5(::std::move(gmock_p5)), \
+ p6(::std::move(gmock_p6))
+#define GMOCK_INTERNAL_INIT_AND_8_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, p7) \
+ (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2, \
+ p3##_type gmock_p3, p4##_type gmock_p4, p5##_type gmock_p5, \
+ p6##_type gmock_p6, p7##_type gmock_p7) \
+ : p0(::std::move(gmock_p0)), \
+ p1(::std::move(gmock_p1)), \
+ p2(::std::move(gmock_p2)), \
+ p3(::std::move(gmock_p3)), \
+ p4(::std::move(gmock_p4)), \
+ p5(::std::move(gmock_p5)), \
+ p6(::std::move(gmock_p6)), \
p7(::std::move(gmock_p7))
-#define GMOCK_INTERNAL_INIT_AND_9_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, \
- p7, p8)\
- (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2, \
- p3##_type gmock_p3, p4##_type gmock_p4, p5##_type gmock_p5, \
- p6##_type gmock_p6, p7##_type gmock_p7, \
- p8##_type gmock_p8) : p0(::std::move(gmock_p0)), \
- p1(::std::move(gmock_p1)), p2(::std::move(gmock_p2)), \
- p3(::std::move(gmock_p3)), p4(::std::move(gmock_p4)), \
- p5(::std::move(gmock_p5)), p6(::std::move(gmock_p6)), \
- p7(::std::move(gmock_p7)), p8(::std::move(gmock_p8))
+#define GMOCK_INTERNAL_INIT_AND_9_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, p7, \
+ p8) \
+ (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2, \
+ p3##_type gmock_p3, p4##_type gmock_p4, p5##_type gmock_p5, \
+ p6##_type gmock_p6, p7##_type gmock_p7, p8##_type gmock_p8) \
+ : p0(::std::move(gmock_p0)), \
+ p1(::std::move(gmock_p1)), \
+ p2(::std::move(gmock_p2)), \
+ p3(::std::move(gmock_p3)), \
+ p4(::std::move(gmock_p4)), \
+ p5(::std::move(gmock_p5)), \
+ p6(::std::move(gmock_p6)), \
+ p7(::std::move(gmock_p7)), \
+ p8(::std::move(gmock_p8))
#define GMOCK_INTERNAL_INIT_AND_10_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, \
- p7, p8, p9)\
- (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2, \
- p3##_type gmock_p3, p4##_type gmock_p4, p5##_type gmock_p5, \
- p6##_type gmock_p6, p7##_type gmock_p7, p8##_type gmock_p8, \
- p9##_type gmock_p9) : p0(::std::move(gmock_p0)), \
- p1(::std::move(gmock_p1)), p2(::std::move(gmock_p2)), \
- p3(::std::move(gmock_p3)), p4(::std::move(gmock_p4)), \
- p5(::std::move(gmock_p5)), p6(::std::move(gmock_p6)), \
- p7(::std::move(gmock_p7)), p8(::std::move(gmock_p8)), \
+ p7, p8, p9) \
+ (p0##_type gmock_p0, p1##_type gmock_p1, p2##_type gmock_p2, \
+ p3##_type gmock_p3, p4##_type gmock_p4, p5##_type gmock_p5, \
+ p6##_type gmock_p6, p7##_type gmock_p7, p8##_type gmock_p8, \
+ p9##_type gmock_p9) \
+ : p0(::std::move(gmock_p0)), \
+ p1(::std::move(gmock_p1)), \
+ p2(::std::move(gmock_p2)), \
+ p3(::std::move(gmock_p3)), \
+ p4(::std::move(gmock_p4)), \
+ p5(::std::move(gmock_p5)), \
+ p6(::std::move(gmock_p6)), \
+ p7(::std::move(gmock_p7)), \
+ p8(::std::move(gmock_p8)), \
p9(::std::move(gmock_p9))
// Defines the copy constructor
#define GMOCK_INTERNAL_DEFN_COPY_AND_0_VALUE_PARAMS() \
- {} // Avoid https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82134
+ {} // Avoid https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82134
#define GMOCK_INTERNAL_DEFN_COPY_AND_1_VALUE_PARAMS(...) = default;
#define GMOCK_INTERNAL_DEFN_COPY_AND_2_VALUE_PARAMS(...) = default;
#define GMOCK_INTERNAL_DEFN_COPY_AND_3_VALUE_PARAMS(...) = default;
@@ -307,30 +344,71 @@
// Declares the fields for storing the value parameters.
#define GMOCK_INTERNAL_DEFN_AND_0_VALUE_PARAMS()
#define GMOCK_INTERNAL_DEFN_AND_1_VALUE_PARAMS(p0) p0##_type p0;
-#define GMOCK_INTERNAL_DEFN_AND_2_VALUE_PARAMS(p0, p1) p0##_type p0; \
- p1##_type p1;
-#define GMOCK_INTERNAL_DEFN_AND_3_VALUE_PARAMS(p0, p1, p2) p0##_type p0; \
- p1##_type p1; p2##_type p2;
-#define GMOCK_INTERNAL_DEFN_AND_4_VALUE_PARAMS(p0, p1, p2, p3) p0##_type p0; \
- p1##_type p1; p2##_type p2; p3##_type p3;
-#define GMOCK_INTERNAL_DEFN_AND_5_VALUE_PARAMS(p0, p1, p2, p3, \
- p4) p0##_type p0; p1##_type p1; p2##_type p2; p3##_type p3; p4##_type p4;
-#define GMOCK_INTERNAL_DEFN_AND_6_VALUE_PARAMS(p0, p1, p2, p3, p4, \
- p5) p0##_type p0; p1##_type p1; p2##_type p2; p3##_type p3; p4##_type p4; \
- p5##_type p5;
-#define GMOCK_INTERNAL_DEFN_AND_7_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, \
- p6) p0##_type p0; p1##_type p1; p2##_type p2; p3##_type p3; p4##_type p4; \
- p5##_type p5; p6##_type p6;
-#define GMOCK_INTERNAL_DEFN_AND_8_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, \
- p7) p0##_type p0; p1##_type p1; p2##_type p2; p3##_type p3; p4##_type p4; \
- p5##_type p5; p6##_type p6; p7##_type p7;
-#define GMOCK_INTERNAL_DEFN_AND_9_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, \
- p7, p8) p0##_type p0; p1##_type p1; p2##_type p2; p3##_type p3; \
- p4##_type p4; p5##_type p5; p6##_type p6; p7##_type p7; p8##_type p8;
+#define GMOCK_INTERNAL_DEFN_AND_2_VALUE_PARAMS(p0, p1) \
+ p0##_type p0; \
+ p1##_type p1;
+#define GMOCK_INTERNAL_DEFN_AND_3_VALUE_PARAMS(p0, p1, p2) \
+ p0##_type p0; \
+ p1##_type p1; \
+ p2##_type p2;
+#define GMOCK_INTERNAL_DEFN_AND_4_VALUE_PARAMS(p0, p1, p2, p3) \
+ p0##_type p0; \
+ p1##_type p1; \
+ p2##_type p2; \
+ p3##_type p3;
+#define GMOCK_INTERNAL_DEFN_AND_5_VALUE_PARAMS(p0, p1, p2, p3, p4) \
+ p0##_type p0; \
+ p1##_type p1; \
+ p2##_type p2; \
+ p3##_type p3; \
+ p4##_type p4;
+#define GMOCK_INTERNAL_DEFN_AND_6_VALUE_PARAMS(p0, p1, p2, p3, p4, p5) \
+ p0##_type p0; \
+ p1##_type p1; \
+ p2##_type p2; \
+ p3##_type p3; \
+ p4##_type p4; \
+ p5##_type p5;
+#define GMOCK_INTERNAL_DEFN_AND_7_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6) \
+ p0##_type p0; \
+ p1##_type p1; \
+ p2##_type p2; \
+ p3##_type p3; \
+ p4##_type p4; \
+ p5##_type p5; \
+ p6##_type p6;
+#define GMOCK_INTERNAL_DEFN_AND_8_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, p7) \
+ p0##_type p0; \
+ p1##_type p1; \
+ p2##_type p2; \
+ p3##_type p3; \
+ p4##_type p4; \
+ p5##_type p5; \
+ p6##_type p6; \
+ p7##_type p7;
+#define GMOCK_INTERNAL_DEFN_AND_9_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, p7, \
+ p8) \
+ p0##_type p0; \
+ p1##_type p1; \
+ p2##_type p2; \
+ p3##_type p3; \
+ p4##_type p4; \
+ p5##_type p5; \
+ p6##_type p6; \
+ p7##_type p7; \
+ p8##_type p8;
#define GMOCK_INTERNAL_DEFN_AND_10_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, \
- p7, p8, p9) p0##_type p0; p1##_type p1; p2##_type p2; p3##_type p3; \
- p4##_type p4; p5##_type p5; p6##_type p6; p7##_type p7; p8##_type p8; \
- p9##_type p9;
+ p7, p8, p9) \
+ p0##_type p0; \
+ p1##_type p1; \
+ p2##_type p2; \
+ p3##_type p3; \
+ p4##_type p4; \
+ p5##_type p5; \
+ p6##_type p6; \
+ p7##_type p7; \
+ p8##_type p8; \
+ p9##_type p9;
// Lists the value parameters.
#define GMOCK_INTERNAL_LIST_AND_0_VALUE_PARAMS()
@@ -338,72 +416,78 @@
#define GMOCK_INTERNAL_LIST_AND_2_VALUE_PARAMS(p0, p1) p0, p1
#define GMOCK_INTERNAL_LIST_AND_3_VALUE_PARAMS(p0, p1, p2) p0, p1, p2
#define GMOCK_INTERNAL_LIST_AND_4_VALUE_PARAMS(p0, p1, p2, p3) p0, p1, p2, p3
-#define GMOCK_INTERNAL_LIST_AND_5_VALUE_PARAMS(p0, p1, p2, p3, p4) p0, p1, \
- p2, p3, p4
-#define GMOCK_INTERNAL_LIST_AND_6_VALUE_PARAMS(p0, p1, p2, p3, p4, p5) p0, \
- p1, p2, p3, p4, p5
-#define GMOCK_INTERNAL_LIST_AND_7_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, \
- p6) p0, p1, p2, p3, p4, p5, p6
-#define GMOCK_INTERNAL_LIST_AND_8_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, \
- p7) p0, p1, p2, p3, p4, p5, p6, p7
-#define GMOCK_INTERNAL_LIST_AND_9_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, \
- p7, p8) p0, p1, p2, p3, p4, p5, p6, p7, p8
+#define GMOCK_INTERNAL_LIST_AND_5_VALUE_PARAMS(p0, p1, p2, p3, p4) \
+ p0, p1, p2, p3, p4
+#define GMOCK_INTERNAL_LIST_AND_6_VALUE_PARAMS(p0, p1, p2, p3, p4, p5) \
+ p0, p1, p2, p3, p4, p5
+#define GMOCK_INTERNAL_LIST_AND_7_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6) \
+ p0, p1, p2, p3, p4, p5, p6
+#define GMOCK_INTERNAL_LIST_AND_8_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, p7) \
+ p0, p1, p2, p3, p4, p5, p6, p7
+#define GMOCK_INTERNAL_LIST_AND_9_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, p7, \
+ p8) \
+ p0, p1, p2, p3, p4, p5, p6, p7, p8
#define GMOCK_INTERNAL_LIST_AND_10_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, \
- p7, p8, p9) p0, p1, p2, p3, p4, p5, p6, p7, p8, p9
+ p7, p8, p9) \
+ p0, p1, p2, p3, p4, p5, p6, p7, p8, p9
// Lists the value parameter types.
#define GMOCK_INTERNAL_LIST_TYPE_AND_0_VALUE_PARAMS()
#define GMOCK_INTERNAL_LIST_TYPE_AND_1_VALUE_PARAMS(p0) , p0##_type
-#define GMOCK_INTERNAL_LIST_TYPE_AND_2_VALUE_PARAMS(p0, p1) , p0##_type, \
- p1##_type
-#define GMOCK_INTERNAL_LIST_TYPE_AND_3_VALUE_PARAMS(p0, p1, p2) , p0##_type, \
- p1##_type, p2##_type
-#define GMOCK_INTERNAL_LIST_TYPE_AND_4_VALUE_PARAMS(p0, p1, p2, p3) , \
- p0##_type, p1##_type, p2##_type, p3##_type
-#define GMOCK_INTERNAL_LIST_TYPE_AND_5_VALUE_PARAMS(p0, p1, p2, p3, p4) , \
- p0##_type, p1##_type, p2##_type, p3##_type, p4##_type
-#define GMOCK_INTERNAL_LIST_TYPE_AND_6_VALUE_PARAMS(p0, p1, p2, p3, p4, p5) , \
- p0##_type, p1##_type, p2##_type, p3##_type, p4##_type, p5##_type
+#define GMOCK_INTERNAL_LIST_TYPE_AND_2_VALUE_PARAMS(p0, p1) \
+ , p0##_type, p1##_type
+#define GMOCK_INTERNAL_LIST_TYPE_AND_3_VALUE_PARAMS(p0, p1, p2) \
+ , p0##_type, p1##_type, p2##_type
+#define GMOCK_INTERNAL_LIST_TYPE_AND_4_VALUE_PARAMS(p0, p1, p2, p3) \
+ , p0##_type, p1##_type, p2##_type, p3##_type
+#define GMOCK_INTERNAL_LIST_TYPE_AND_5_VALUE_PARAMS(p0, p1, p2, p3, p4) \
+ , p0##_type, p1##_type, p2##_type, p3##_type, p4##_type
+#define GMOCK_INTERNAL_LIST_TYPE_AND_6_VALUE_PARAMS(p0, p1, p2, p3, p4, p5) \
+ , p0##_type, p1##_type, p2##_type, p3##_type, p4##_type, p5##_type
#define GMOCK_INTERNAL_LIST_TYPE_AND_7_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, \
- p6) , p0##_type, p1##_type, p2##_type, p3##_type, p4##_type, p5##_type, \
- p6##_type
+ p6) \
+ , p0##_type, p1##_type, p2##_type, p3##_type, p4##_type, p5##_type, p6##_type
#define GMOCK_INTERNAL_LIST_TYPE_AND_8_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, \
- p6, p7) , p0##_type, p1##_type, p2##_type, p3##_type, p4##_type, \
- p5##_type, p6##_type, p7##_type
+ p6, p7) \
+ , p0##_type, p1##_type, p2##_type, p3##_type, p4##_type, p5##_type, \
+ p6##_type, p7##_type
#define GMOCK_INTERNAL_LIST_TYPE_AND_9_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, \
- p6, p7, p8) , p0##_type, p1##_type, p2##_type, p3##_type, p4##_type, \
- p5##_type, p6##_type, p7##_type, p8##_type
+ p6, p7, p8) \
+ , p0##_type, p1##_type, p2##_type, p3##_type, p4##_type, p5##_type, \
+ p6##_type, p7##_type, p8##_type
#define GMOCK_INTERNAL_LIST_TYPE_AND_10_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, \
- p6, p7, p8, p9) , p0##_type, p1##_type, p2##_type, p3##_type, p4##_type, \
- p5##_type, p6##_type, p7##_type, p8##_type, p9##_type
+ p6, p7, p8, p9) \
+ , p0##_type, p1##_type, p2##_type, p3##_type, p4##_type, p5##_type, \
+ p6##_type, p7##_type, p8##_type, p9##_type
// Declares the value parameters.
#define GMOCK_INTERNAL_DECL_AND_0_VALUE_PARAMS()
#define GMOCK_INTERNAL_DECL_AND_1_VALUE_PARAMS(p0) p0##_type p0
-#define GMOCK_INTERNAL_DECL_AND_2_VALUE_PARAMS(p0, p1) p0##_type p0, \
- p1##_type p1
-#define GMOCK_INTERNAL_DECL_AND_3_VALUE_PARAMS(p0, p1, p2) p0##_type p0, \
- p1##_type p1, p2##_type p2
-#define GMOCK_INTERNAL_DECL_AND_4_VALUE_PARAMS(p0, p1, p2, p3) p0##_type p0, \
- p1##_type p1, p2##_type p2, p3##_type p3
-#define GMOCK_INTERNAL_DECL_AND_5_VALUE_PARAMS(p0, p1, p2, p3, \
- p4) p0##_type p0, p1##_type p1, p2##_type p2, p3##_type p3, p4##_type p4
-#define GMOCK_INTERNAL_DECL_AND_6_VALUE_PARAMS(p0, p1, p2, p3, p4, \
- p5) p0##_type p0, p1##_type p1, p2##_type p2, p3##_type p3, p4##_type p4, \
- p5##_type p5
-#define GMOCK_INTERNAL_DECL_AND_7_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, \
- p6) p0##_type p0, p1##_type p1, p2##_type p2, p3##_type p3, p4##_type p4, \
- p5##_type p5, p6##_type p6
-#define GMOCK_INTERNAL_DECL_AND_8_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, \
- p7) p0##_type p0, p1##_type p1, p2##_type p2, p3##_type p3, p4##_type p4, \
- p5##_type p5, p6##_type p6, p7##_type p7
-#define GMOCK_INTERNAL_DECL_AND_9_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, \
- p7, p8) p0##_type p0, p1##_type p1, p2##_type p2, p3##_type p3, \
- p4##_type p4, p5##_type p5, p6##_type p6, p7##_type p7, p8##_type p8
+#define GMOCK_INTERNAL_DECL_AND_2_VALUE_PARAMS(p0, p1) \
+ p0##_type p0, p1##_type p1
+#define GMOCK_INTERNAL_DECL_AND_3_VALUE_PARAMS(p0, p1, p2) \
+ p0##_type p0, p1##_type p1, p2##_type p2
+#define GMOCK_INTERNAL_DECL_AND_4_VALUE_PARAMS(p0, p1, p2, p3) \
+ p0##_type p0, p1##_type p1, p2##_type p2, p3##_type p3
+#define GMOCK_INTERNAL_DECL_AND_5_VALUE_PARAMS(p0, p1, p2, p3, p4) \
+ p0##_type p0, p1##_type p1, p2##_type p2, p3##_type p3, p4##_type p4
+#define GMOCK_INTERNAL_DECL_AND_6_VALUE_PARAMS(p0, p1, p2, p3, p4, p5) \
+ p0##_type p0, p1##_type p1, p2##_type p2, p3##_type p3, p4##_type p4, \
+ p5##_type p5
+#define GMOCK_INTERNAL_DECL_AND_7_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6) \
+ p0##_type p0, p1##_type p1, p2##_type p2, p3##_type p3, p4##_type p4, \
+ p5##_type p5, p6##_type p6
+#define GMOCK_INTERNAL_DECL_AND_8_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, p7) \
+ p0##_type p0, p1##_type p1, p2##_type p2, p3##_type p3, p4##_type p4, \
+ p5##_type p5, p6##_type p6, p7##_type p7
+#define GMOCK_INTERNAL_DECL_AND_9_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, p7, \
+ p8) \
+ p0##_type p0, p1##_type p1, p2##_type p2, p3##_type p3, p4##_type p4, \
+ p5##_type p5, p6##_type p6, p7##_type p7, p8##_type p8
#define GMOCK_INTERNAL_DECL_AND_10_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, \
- p7, p8, p9) p0##_type p0, p1##_type p1, p2##_type p2, p3##_type p3, \
- p4##_type p4, p5##_type p5, p6##_type p6, p7##_type p7, p8##_type p8, \
- p9##_type p9
+ p7, p8, p9) \
+ p0##_type p0, p1##_type p1, p2##_type p2, p3##_type p3, p4##_type p4, \
+ p5##_type p5, p6##_type p6, p7##_type p7, p8##_type p8, p9##_type p9
// The suffix of the class template implementing the action template.
#define GMOCK_INTERNAL_COUNT_AND_0_VALUE_PARAMS()
@@ -415,40 +499,43 @@
#define GMOCK_INTERNAL_COUNT_AND_6_VALUE_PARAMS(p0, p1, p2, p3, p4, p5) P6
#define GMOCK_INTERNAL_COUNT_AND_7_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6) P7
#define GMOCK_INTERNAL_COUNT_AND_8_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, \
- p7) P8
+ p7) \
+ P8
#define GMOCK_INTERNAL_COUNT_AND_9_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, \
- p7, p8) P9
+ p7, p8) \
+ P9
#define GMOCK_INTERNAL_COUNT_AND_10_VALUE_PARAMS(p0, p1, p2, p3, p4, p5, p6, \
- p7, p8, p9) P10
+ p7, p8, p9) \
+ P10
// The name of the class template implementing the action template.
-#define GMOCK_ACTION_CLASS_(name, value_params)\
- GTEST_CONCAT_TOKEN_(name##Action, GMOCK_INTERNAL_COUNT_##value_params)
+#define GMOCK_ACTION_CLASS_(name, value_params) \
+ GTEST_CONCAT_TOKEN_(name##Action, GMOCK_INTERNAL_COUNT_##value_params)
#define ACTION_TEMPLATE(name, template_params, value_params) \
template <GMOCK_INTERNAL_DECL_##template_params \
- GMOCK_INTERNAL_DECL_TYPE_##value_params> \
+ GMOCK_INTERNAL_DECL_TYPE_##value_params> \
class GMOCK_ACTION_CLASS_(name, value_params) { \
public: \
explicit GMOCK_ACTION_CLASS_(name, value_params)( \
GMOCK_INTERNAL_DECL_##value_params) \
GMOCK_PP_IF(GMOCK_PP_IS_EMPTY(GMOCK_INTERNAL_COUNT_##value_params), \
- = default; , \
+ = default; \
+ , \
: impl_(std::make_shared<gmock_Impl>( \
- GMOCK_INTERNAL_LIST_##value_params)) { }) \
- GMOCK_ACTION_CLASS_(name, value_params)( \
- const GMOCK_ACTION_CLASS_(name, value_params)&) noexcept \
- GMOCK_INTERNAL_DEFN_COPY_##value_params \
- GMOCK_ACTION_CLASS_(name, value_params)( \
- GMOCK_ACTION_CLASS_(name, value_params)&&) noexcept \
- GMOCK_INTERNAL_DEFN_COPY_##value_params \
- template <typename F> \
- operator ::testing::Action<F>() const { \
+ GMOCK_INTERNAL_LIST_##value_params)){}) \
+ GMOCK_ACTION_CLASS_(name, value_params)(const GMOCK_ACTION_CLASS_( \
+ name, value_params) &) noexcept GMOCK_INTERNAL_DEFN_COPY_ \
+ ##value_params GMOCK_ACTION_CLASS_(name, value_params)( \
+ GMOCK_ACTION_CLASS_(name, value_params) &&) noexcept \
+ GMOCK_INTERNAL_DEFN_COPY_##value_params template <typename F> \
+ operator ::testing::Action<F>() const { \
return GMOCK_PP_IF( \
GMOCK_PP_IS_EMPTY(GMOCK_INTERNAL_COUNT_##value_params), \
- (::testing::internal::MakeAction<F, gmock_Impl>()), \
- (::testing::internal::MakeAction<F>(impl_))); \
+ (::testing::internal::MakeAction<F, gmock_Impl>()), \
+ (::testing::internal::MakeAction<F>(impl_))); \
} \
+ \
private: \
class gmock_Impl { \
public: \
@@ -458,34 +545,35 @@
return_type gmock_PerformImpl(GMOCK_ACTION_ARG_TYPES_AND_NAMES_) const; \
GMOCK_INTERNAL_DEFN_##value_params \
}; \
- GMOCK_PP_IF(GMOCK_PP_IS_EMPTY(GMOCK_INTERNAL_COUNT_##value_params), \
- , std::shared_ptr<const gmock_Impl> impl_;) \
+ GMOCK_PP_IF(GMOCK_PP_IS_EMPTY(GMOCK_INTERNAL_COUNT_##value_params), , \
+ std::shared_ptr<const gmock_Impl> impl_;) \
}; \
template <GMOCK_INTERNAL_DECL_##template_params \
- GMOCK_INTERNAL_DECL_TYPE_##value_params> \
- GMOCK_ACTION_CLASS_(name, value_params)< \
- GMOCK_INTERNAL_LIST_##template_params \
- GMOCK_INTERNAL_LIST_TYPE_##value_params> name( \
- GMOCK_INTERNAL_DECL_##value_params) GTEST_MUST_USE_RESULT_; \
+ GMOCK_INTERNAL_DECL_TYPE_##value_params> \
+ GMOCK_ACTION_CLASS_( \
+ name, value_params)<GMOCK_INTERNAL_LIST_##template_params \
+ GMOCK_INTERNAL_LIST_TYPE_##value_params> \
+ name(GMOCK_INTERNAL_DECL_##value_params) GTEST_MUST_USE_RESULT_; \
template <GMOCK_INTERNAL_DECL_##template_params \
- GMOCK_INTERNAL_DECL_TYPE_##value_params> \
- inline GMOCK_ACTION_CLASS_(name, value_params)< \
- GMOCK_INTERNAL_LIST_##template_params \
- GMOCK_INTERNAL_LIST_TYPE_##value_params> name( \
- GMOCK_INTERNAL_DECL_##value_params) { \
- return GMOCK_ACTION_CLASS_(name, value_params)< \
- GMOCK_INTERNAL_LIST_##template_params \
- GMOCK_INTERNAL_LIST_TYPE_##value_params>( \
- GMOCK_INTERNAL_LIST_##value_params); \
+ GMOCK_INTERNAL_DECL_TYPE_##value_params> \
+ inline GMOCK_ACTION_CLASS_( \
+ name, value_params)<GMOCK_INTERNAL_LIST_##template_params \
+ GMOCK_INTERNAL_LIST_TYPE_##value_params> \
+ name(GMOCK_INTERNAL_DECL_##value_params) { \
+ return GMOCK_ACTION_CLASS_( \
+ name, value_params)<GMOCK_INTERNAL_LIST_##template_params \
+ GMOCK_INTERNAL_LIST_TYPE_##value_params>( \
+ GMOCK_INTERNAL_LIST_##value_params); \
} \
template <GMOCK_INTERNAL_DECL_##template_params \
- GMOCK_INTERNAL_DECL_TYPE_##value_params> \
+ GMOCK_INTERNAL_DECL_TYPE_##value_params> \
template <typename function_type, typename return_type, typename args_type, \
GMOCK_ACTION_TEMPLATE_ARGS_NAMES_> \
- return_type GMOCK_ACTION_CLASS_(name, value_params)< \
- GMOCK_INTERNAL_LIST_##template_params \
- GMOCK_INTERNAL_LIST_TYPE_##value_params>::gmock_Impl::gmock_PerformImpl( \
- GMOCK_ACTION_ARG_TYPES_AND_NAMES_UNUSED_) const
+ return_type GMOCK_ACTION_CLASS_( \
+ name, value_params)<GMOCK_INTERNAL_LIST_##template_params \
+ GMOCK_INTERNAL_LIST_TYPE_##value_params>:: \
+ gmock_Impl::gmock_PerformImpl(GMOCK_ACTION_ARG_TYPES_AND_NAMES_UNUSED_) \
+ const
namespace testing {
@@ -495,8 +583,8 @@
// is expanded and macro expansion cannot contain #pragma. Therefore
// we suppress them here.
#ifdef _MSC_VER
-# pragma warning(push)
-# pragma warning(disable:4100)
+#pragma warning(push)
+#pragma warning(disable : 4100)
#endif
namespace internal {
@@ -512,7 +600,8 @@
template <std::size_t index, typename... Params>
struct InvokeArgumentAction {
- template <typename... Args>
+ template <typename... Args,
+ typename = typename std::enable_if<(index < sizeof...(Args))>::type>
auto operator()(Args&&... args) const -> decltype(internal::InvokeArgument(
std::get<index>(std::forward_as_tuple(std::forward<Args>(args)...)),
std::declval<const Params&>()...)) {
@@ -565,7 +654,7 @@
}
#ifdef _MSC_VER
-# pragma warning(pop)
+#pragma warning(pop)
#endif
} // namespace testing
diff --git a/third_party/googletest/src/googlemock/include/gmock/gmock-more-matchers.h b/third_party/googletest/src/googlemock/include/gmock/gmock-more-matchers.h
index dfc77e3..47aaf98 100644
--- a/third_party/googletest/src/googlemock/include/gmock/gmock-more-matchers.h
+++ b/third_party/googletest/src/googlemock/include/gmock/gmock-more-matchers.h
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Google Mock - a framework for writing C++ mock classes.
//
// This file implements some matchers that depend on gmock-matchers.h.
@@ -35,7 +34,8 @@
// Note that tests are implemented in gmock-matchers_test.cc rather than
// gmock-more-matchers-test.cc.
-// GOOGLETEST_CM0002 DO NOT DELETE
+// IWYU pragma: private, include "gmock/gmock.h"
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_MORE_MATCHERS_H_
#define GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_MORE_MATCHERS_H_
@@ -47,13 +47,13 @@
// Silence C4100 (unreferenced formal
// parameter) for MSVC
#ifdef _MSC_VER
-# pragma warning(push)
-# pragma warning(disable:4100)
+#pragma warning(push)
+#pragma warning(disable : 4100)
#if (_MSC_VER == 1900)
// and silence C4800 (C4800: 'int *const ': forcing value
// to bool 'true' or 'false') for MSVC 14
-# pragma warning(disable:4800)
- #endif
+#pragma warning(disable : 4800)
+#endif
#endif
// Defines a matcher that matches an empty container. The container must
@@ -83,10 +83,9 @@
}
#ifdef _MSC_VER
-# pragma warning(pop)
+#pragma warning(pop)
#endif
-
} // namespace testing
#endif // GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_MORE_MATCHERS_H_
diff --git a/third_party/googletest/src/googlemock/include/gmock/gmock-nice-strict.h b/third_party/googletest/src/googlemock/include/gmock/gmock-nice-strict.h
index b03b770..4f0eb35 100644
--- a/third_party/googletest/src/googlemock/include/gmock/gmock-nice-strict.h
+++ b/third_party/googletest/src/googlemock/include/gmock/gmock-nice-strict.h
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Implements class templates NiceMock, NaggyMock, and StrictMock.
//
// Given a mock class MockFoo that is created using Google Mock,
@@ -58,11 +57,13 @@
// In particular, nesting NiceMock, NaggyMock, and StrictMock is NOT
// supported.
-// GOOGLETEST_CM0002 DO NOT DELETE
+// IWYU pragma: private, include "gmock/gmock.h"
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_NICE_STRICT_H_
#define GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_NICE_STRICT_H_
+#include <cstdint>
#include <type_traits>
#include "gmock/gmock-spec-builders.h"
@@ -109,25 +110,37 @@
template <typename Base>
class NiceMockImpl {
public:
- NiceMockImpl() { ::testing::Mock::AllowUninterestingCalls(this); }
+ NiceMockImpl() {
+ ::testing::Mock::AllowUninterestingCalls(reinterpret_cast<uintptr_t>(this));
+ }
- ~NiceMockImpl() { ::testing::Mock::UnregisterCallReaction(this); }
+ ~NiceMockImpl() {
+ ::testing::Mock::UnregisterCallReaction(reinterpret_cast<uintptr_t>(this));
+ }
};
template <typename Base>
class NaggyMockImpl {
public:
- NaggyMockImpl() { ::testing::Mock::WarnUninterestingCalls(this); }
+ NaggyMockImpl() {
+ ::testing::Mock::WarnUninterestingCalls(reinterpret_cast<uintptr_t>(this));
+ }
- ~NaggyMockImpl() { ::testing::Mock::UnregisterCallReaction(this); }
+ ~NaggyMockImpl() {
+ ::testing::Mock::UnregisterCallReaction(reinterpret_cast<uintptr_t>(this));
+ }
};
template <typename Base>
class StrictMockImpl {
public:
- StrictMockImpl() { ::testing::Mock::FailUninterestingCalls(this); }
+ StrictMockImpl() {
+ ::testing::Mock::FailUninterestingCalls(reinterpret_cast<uintptr_t>(this));
+ }
- ~StrictMockImpl() { ::testing::Mock::UnregisterCallReaction(this); }
+ ~StrictMockImpl() {
+ ::testing::Mock::UnregisterCallReaction(reinterpret_cast<uintptr_t>(this));
+ }
};
} // namespace internal
@@ -169,7 +182,8 @@
}
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(NiceMock);
+ NiceMock(const NiceMock&) = delete;
+ NiceMock& operator=(const NiceMock&) = delete;
};
template <class MockClass>
@@ -210,7 +224,8 @@
}
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(NaggyMock);
+ NaggyMock(const NaggyMock&) = delete;
+ NaggyMock& operator=(const NaggyMock&) = delete;
};
template <class MockClass>
@@ -251,7 +266,8 @@
}
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(StrictMock);
+ StrictMock(const StrictMock&) = delete;
+ StrictMock& operator=(const StrictMock&) = delete;
};
#undef GTEST_INTERNAL_EMPTY_BASE_CLASS
diff --git a/third_party/googletest/src/googlemock/include/gmock/gmock-spec-builders.h b/third_party/googletest/src/googlemock/include/gmock/gmock-spec-builders.h
index 41323c1..45cc605 100644
--- a/third_party/googletest/src/googlemock/include/gmock/gmock-spec-builders.h
+++ b/third_party/googletest/src/googlemock/include/gmock/gmock-spec-builders.h
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Google Mock - a framework for writing C++ mock classes.
//
// This file implements the ON_CALL() and EXPECT_CALL() macros.
@@ -56,11 +55,13 @@
// where all clauses are optional, and .InSequence()/.After()/
// .WillOnce() can appear any number of times.
-// GOOGLETEST_CM0002 DO NOT DELETE
+// IWYU pragma: private, include "gmock/gmock.h"
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_SPEC_BUILDERS_H_
#define GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_SPEC_BUILDERS_H_
+#include <cstdint>
#include <functional>
#include <map>
#include <memory>
@@ -70,6 +71,7 @@
#include <type_traits>
#include <utility>
#include <vector>
+
#include "gmock/gmock-actions.h"
#include "gmock/gmock-cardinalities.h"
#include "gmock/gmock-matchers.h"
@@ -78,7 +80,7 @@
#include "gtest/gtest.h"
#if GTEST_HAS_EXCEPTIONS
-# include <stdexcept> // NOLINT
+#include <stdexcept> // NOLINT
#endif
GTEST_DISABLE_MSC_WARNINGS_PUSH_(4251 \
@@ -97,13 +99,15 @@
namespace internal {
// Implements a mock function.
-template <typename F> class FunctionMocker;
+template <typename F>
+class FunctionMocker;
// Base class for expectations.
class ExpectationBase;
// Implements an expectation.
-template <typename F> class TypedExpectation;
+template <typename F>
+class TypedExpectation;
// Helper class for testing the Expectation class template.
class ExpectationTester;
@@ -129,9 +133,6 @@
// calls to ensure the integrity of the mock objects' states.
GTEST_API_ GTEST_DECLARE_STATIC_MUTEX_(g_gmock_mutex);
-// Untyped base class for ActionResultHolder<R>.
-class UntypedActionResultHolderBase;
-
// Abstract base class of FunctionMocker. This is the
// type-agnostic part of the function mocker interface. Its pure
// virtual methods are implemented by FunctionMocker.
@@ -154,27 +155,12 @@
// responsibility to guarantee the correctness of the arguments'
// types.
- // Performs the default action with the given arguments and returns
- // the action's result. The call description string will be used in
- // the error message to describe the call in the case the default
- // action fails.
- // L = *
- virtual UntypedActionResultHolderBase* UntypedPerformDefaultAction(
- void* untyped_args, const std::string& call_description) const = 0;
-
- // Performs the given action with the given arguments and returns
- // the action's result.
- // L = *
- virtual UntypedActionResultHolderBase* UntypedPerformAction(
- const void* untyped_action, void* untyped_args) const = 0;
-
// Writes a message that the call is uninteresting (i.e. neither
// explicitly expected nor explicitly unexpected) to the given
// ostream.
- virtual void UntypedDescribeUninterestingCall(
- const void* untyped_args,
- ::std::ostream* os) const
- GTEST_LOCK_EXCLUDED_(g_gmock_mutex) = 0;
+ virtual void UntypedDescribeUninterestingCall(const void* untyped_args,
+ ::std::ostream* os) const
+ GTEST_LOCK_EXCLUDED_(g_gmock_mutex) = 0;
// Returns the expectation that matches the given function arguments
// (or NULL is there's no match); when a match is found,
@@ -183,10 +169,9 @@
// is_excessive is modified to indicate whether the call exceeds the
// expected number.
virtual const ExpectationBase* UntypedFindMatchingExpectation(
- const void* untyped_args,
- const void** untyped_action, bool* is_excessive,
+ const void* untyped_args, const void** untyped_action, bool* is_excessive,
::std::ostream* what, ::std::ostream* why)
- GTEST_LOCK_EXCLUDED_(g_gmock_mutex) = 0;
+ GTEST_LOCK_EXCLUDED_(g_gmock_mutex) = 0;
// Prints the given function arguments to the ostream.
virtual void UntypedPrintArgs(const void* untyped_args,
@@ -196,8 +181,7 @@
// this information in the global mock registry. Will be called
// whenever an EXPECT_CALL() or ON_CALL() is executed on this mock
// method.
- void RegisterOwner(const void* mock_obj)
- GTEST_LOCK_EXCLUDED_(g_gmock_mutex);
+ void RegisterOwner(const void* mock_obj) GTEST_LOCK_EXCLUDED_(g_gmock_mutex);
// Sets the mock object this mock method belongs to, and sets the
// name of the mock function. Will be called upon each invocation
@@ -208,20 +192,11 @@
// Returns the mock object this mock method belongs to. Must be
// called after RegisterOwner() or SetOwnerAndName() has been
// called.
- const void* MockObject() const
- GTEST_LOCK_EXCLUDED_(g_gmock_mutex);
+ const void* MockObject() const GTEST_LOCK_EXCLUDED_(g_gmock_mutex);
// Returns the name of this mock method. Must be called after
// SetOwnerAndName() has been called.
- const char* Name() const
- GTEST_LOCK_EXCLUDED_(g_gmock_mutex);
-
- // Returns the result of invoking this mock function with the given
- // arguments. This function can be safely called from multiple
- // threads concurrently. The caller is responsible for deleting the
- // result.
- UntypedActionResultHolderBase* UntypedInvokeWith(void* untyped_args)
- GTEST_LOCK_EXCLUDED_(g_gmock_mutex);
+ const char* Name() const GTEST_LOCK_EXCLUDED_(g_gmock_mutex);
protected:
typedef std::vector<const void*> UntypedOnCallSpecs;
@@ -430,29 +405,28 @@
// Tells Google Mock to allow uninteresting calls on the given mock
// object.
- static void AllowUninterestingCalls(const void* mock_obj)
+ static void AllowUninterestingCalls(uintptr_t mock_obj)
GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex);
// Tells Google Mock to warn the user about uninteresting calls on
// the given mock object.
- static void WarnUninterestingCalls(const void* mock_obj)
+ static void WarnUninterestingCalls(uintptr_t mock_obj)
GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex);
// Tells Google Mock to fail uninteresting calls on the given mock
// object.
- static void FailUninterestingCalls(const void* mock_obj)
+ static void FailUninterestingCalls(uintptr_t mock_obj)
GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex);
// Tells Google Mock the given mock object is being destroyed and
// its entry in the call-reaction table should be removed.
- static void UnregisterCallReaction(const void* mock_obj)
+ static void UnregisterCallReaction(uintptr_t mock_obj)
GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex);
// Returns the reaction Google Mock will have on uninteresting calls
// made on the given mock object.
static internal::CallReaction GetReactionOnUninterestingCalls(
- const void* mock_obj)
- GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex);
+ const void* mock_obj) GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex);
// Verifies that all expectations on the given mock object have been
// satisfied. Reports one or more Google Test non-fatal failures
@@ -465,17 +439,16 @@
GTEST_EXCLUSIVE_LOCK_REQUIRED_(internal::g_gmock_mutex);
// Registers a mock object and a mock method it owns.
- static void Register(
- const void* mock_obj,
- internal::UntypedFunctionMockerBase* mocker)
- GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex);
+ static void Register(const void* mock_obj,
+ internal::UntypedFunctionMockerBase* mocker)
+ GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex);
// Tells Google Mock where in the source code mock_obj is used in an
// ON_CALL or EXPECT_CALL. In case mock_obj is leaked, this
// information helps the user identify which object it is.
- static void RegisterUseByOnCallOrExpectCall(
- const void* mock_obj, const char* file, int line)
- GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex);
+ static void RegisterUseByOnCallOrExpectCall(const void* mock_obj,
+ const char* file, int line)
+ GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex);
// Unregisters a mock method; removes the owning mock object from
// the registry when the last mock method associated with it has
@@ -632,7 +605,6 @@
Expectation::Set expectations_;
};
-
// Sequence objects are used by a user to specify the relative order
// in which the expectations should match. They are copyable (we rely
// on the compiler-defined copy constructor and assignment operator).
@@ -678,10 +650,12 @@
public:
InSequence();
~InSequence();
+
private:
bool sequence_created_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(InSequence); // NOLINT
+ InSequence(const InSequence&) = delete;
+ InSequence& operator=(const InSequence&) = delete;
} GTEST_ATTRIBUTE_UNUSED_;
namespace internal {
@@ -784,40 +758,34 @@
// the current thread.
// Retires all pre-requisites of this expectation.
- void RetireAllPreRequisites()
- GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex);
+ void RetireAllPreRequisites() GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex);
// Returns true if and only if this expectation is retired.
- bool is_retired() const
- GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
+ bool is_retired() const GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
g_gmock_mutex.AssertHeld();
return retired_;
}
// Retires this expectation.
- void Retire()
- GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
+ void Retire() GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
g_gmock_mutex.AssertHeld();
retired_ = true;
}
// Returns true if and only if this expectation is satisfied.
- bool IsSatisfied() const
- GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
+ bool IsSatisfied() const GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
g_gmock_mutex.AssertHeld();
return cardinality().IsSatisfiedByCallCount(call_count_);
}
// Returns true if and only if this expectation is saturated.
- bool IsSaturated() const
- GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
+ bool IsSaturated() const GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
g_gmock_mutex.AssertHeld();
return cardinality().IsSaturatedByCallCount(call_count_);
}
// Returns true if and only if this expectation is over-saturated.
- bool IsOverSaturated() const
- GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
+ bool IsOverSaturated() const GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
g_gmock_mutex.AssertHeld();
return cardinality().IsOverSaturatedByCallCount(call_count_);
}
@@ -832,15 +800,13 @@
GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex);
// Returns the number this expectation has been invoked.
- int call_count() const
- GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
+ int call_count() const GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
g_gmock_mutex.AssertHeld();
return call_count_;
}
// Increments the number this expectation has been invoked.
- void IncrementCallCount()
- GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
+ void IncrementCallCount() GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
g_gmock_mutex.AssertHeld();
call_count_++;
}
@@ -849,8 +815,7 @@
// WillRepeatedly() clauses) against the cardinality if this hasn't
// been done before. Prints a warning if there are too many or too
// few actions.
- void CheckActionCountIfNotDone() const
- GTEST_LOCK_EXCLUDED_(mutex_);
+ void CheckActionCountIfNotDone() const GTEST_LOCK_EXCLUDED_(mutex_);
friend class ::testing::Sequence;
friend class ::testing::internal::ExpectationTester;
@@ -863,12 +828,12 @@
// This group of fields are part of the spec and won't change after
// an EXPECT_CALL() statement finishes.
- const char* file_; // The file that contains the expectation.
- int line_; // The line number of the expectation.
+ const char* file_; // The file that contains the expectation.
+ int line_; // The line number of the expectation.
const std::string source_text_; // The EXPECT_CALL(...) source text.
// True if and only if the cardinality is specified explicitly.
bool cardinality_specified_;
- Cardinality cardinality_; // The cardinality of the expectation.
+ Cardinality cardinality_; // The cardinality of the expectation.
// The immediate pre-requisites (i.e. expectations that must be
// satisfied before this expectation can be matched) of this
// expectation. We use std::shared_ptr in the set because we want an
@@ -887,12 +852,18 @@
bool retires_on_saturation_;
Clause last_clause_;
mutable bool action_count_checked_; // Under mutex_.
- mutable Mutex mutex_; // Protects action_count_checked_.
-}; // class ExpectationBase
+ mutable Mutex mutex_; // Protects action_count_checked_.
+}; // class ExpectationBase
-// Impements an expectation for the given function type.
template <typename F>
-class TypedExpectation : public ExpectationBase {
+class TypedExpectation;
+
+// Implements an expectation for the given function type.
+template <typename R, typename... Args>
+class TypedExpectation<R(Args...)> : public ExpectationBase {
+ private:
+ using F = R(Args...);
+
public:
typedef typename Function<F>::ArgumentTuple ArgumentTuple;
typedef typename Function<F>::ArgumentMatcherTuple ArgumentMatcherTuple;
@@ -945,9 +916,7 @@
}
// Implements the .Times() clause.
- TypedExpectation& Times(int n) {
- return Times(Exactly(n));
- }
+ TypedExpectation& Times(int n) { return Times(Exactly(n)); }
// Implements the .InSequence() clause.
TypedExpectation& InSequence(const Sequence& s) {
@@ -1007,14 +976,31 @@
return After(s1, s2, s3, s4).After(s5);
}
- // Implements the .WillOnce() clause.
- TypedExpectation& WillOnce(const Action<F>& action) {
+ // Preferred, type-safe overload: consume anything that can be directly
+ // converted to a OnceAction, except for Action<F> objects themselves.
+ TypedExpectation& WillOnce(OnceAction<F> once_action) {
+ // Call the overload below, smuggling the OnceAction as a copyable callable.
+ // We know this is safe because a WillOnce action will not be called more
+ // than once.
+ return WillOnce(Action<F>(ActionAdaptor{
+ std::make_shared<OnceAction<F>>(std::move(once_action)),
+ }));
+ }
+
+ // Fallback overload: accept Action<F> objects and those actions that define
+ // `operator Action<F>` but not `operator OnceAction<F>`.
+ //
+ // This is templated in order to cause the overload above to be preferred
+ // when the input is convertible to either type.
+ template <int&... ExplicitArgumentBarrier, typename = void>
+ TypedExpectation& WillOnce(Action<F> action) {
ExpectSpecProperty(last_clause_ <= kWillOnce,
".WillOnce() cannot appear after "
".WillRepeatedly() or .RetiresOnSaturation().");
last_clause_ = kWillOnce;
- untyped_actions_.push_back(new Action<F>(action));
+ untyped_actions_.push_back(new Action<F>(std::move(action)));
+
if (!cardinality_specified()) {
set_cardinality(Exactly(static_cast<int>(untyped_actions_.size())));
}
@@ -1062,9 +1048,7 @@
// Returns the matchers for the arguments as specified inside the
// EXPECT_CALL() macro.
- const ArgumentMatcherTuple& matchers() const {
- return matchers_;
- }
+ const ArgumentMatcherTuple& matchers() const { return matchers_; }
// Returns the matcher specified by the .With() clause.
const Matcher<const ArgumentTuple&>& extra_matcher() const {
@@ -1088,6 +1072,16 @@
template <typename Function>
friend class FunctionMocker;
+ // An adaptor that turns a OneAction<F> into something compatible with
+ // Action<F>. Must be called at most once.
+ struct ActionAdaptor {
+ std::shared_ptr<OnceAction<R(Args...)>> once_action;
+
+ R operator()(Args&&... args) const {
+ return std::move(*once_action).Call(std::forward<Args>(args)...);
+ }
+ };
+
// Returns an Expectation object that references and co-owns this
// expectation.
Expectation GetHandle() override { return owner_->GetHandleOf(this); }
@@ -1119,10 +1113,8 @@
// Describes the result of matching the arguments against this
// expectation to the given ostream.
- void ExplainMatchResultTo(
- const ArgumentTuple& args,
- ::std::ostream* os) const
- GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
+ void ExplainMatchResultTo(const ArgumentTuple& args, ::std::ostream* os) const
+ GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
g_gmock_mutex.AssertHeld();
if (is_retired()) {
@@ -1181,9 +1173,9 @@
::std::stringstream ss;
DescribeLocationTo(&ss);
ss << "Actions ran out in " << source_text() << "...\n"
- << "Called " << count << " times, but only "
- << action_count << " WillOnce()"
- << (action_count == 1 ? " is" : "s are") << " specified - ";
+ << "Called " << count << " times, but only " << action_count
+ << " WillOnce()" << (action_count == 1 ? " is" : "s are")
+ << " specified - ";
mocker->DescribeDefaultActionTo(args, &ss);
Log(kWarning, ss.str(), 1);
}
@@ -1225,7 +1217,7 @@
}
// Must be done after IncrementCount()!
- *what << "Mock function call matches " << source_text() <<"...\n";
+ *what << "Mock function call matches " << source_text() << "...\n";
return &(GetCurrentAction(mocker, args));
}
@@ -1236,7 +1228,8 @@
Matcher<const ArgumentTuple&> extra_matcher_;
Action<F> repeated_action_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(TypedExpectation);
+ TypedExpectation(const TypedExpectation&) = delete;
+ TypedExpectation& operator=(const TypedExpectation&) = delete;
}; // class TypedExpectation
// A MockSpec object is used by ON_CALL() or EXPECT_CALL() for
@@ -1258,8 +1251,8 @@
class MockSpec {
public:
typedef typename internal::Function<F>::ArgumentTuple ArgumentTuple;
- typedef typename internal::Function<F>::ArgumentMatcherTuple
- ArgumentMatcherTuple;
+ typedef
+ typename internal::Function<F>::ArgumentMatcherTuple ArgumentMatcherTuple;
// Constructs a MockSpec object, given the function mocker object
// that the spec is associated with.
@@ -1269,8 +1262,9 @@
// Adds a new default action spec to the function mocker and returns
// the newly created spec.
- internal::OnCallSpec<F>& InternalDefaultActionSetAt(
- const char* file, int line, const char* obj, const char* call) {
+ internal::OnCallSpec<F>& InternalDefaultActionSetAt(const char* file,
+ int line, const char* obj,
+ const char* call) {
LogWithLocation(internal::kInfo, file, line,
std::string("ON_CALL(") + obj + ", " + call + ") invoked");
return function_mocker_->AddNewOnCallSpec(file, line, matchers_);
@@ -1278,13 +1272,14 @@
// Adds a new expectation spec to the function mocker and returns
// the newly created spec.
- internal::TypedExpectation<F>& InternalExpectedAt(
- const char* file, int line, const char* obj, const char* call) {
+ internal::TypedExpectation<F>& InternalExpectedAt(const char* file, int line,
+ const char* obj,
+ const char* call) {
const std::string source_text(std::string("EXPECT_CALL(") + obj + ", " +
call + ")");
LogWithLocation(internal::kInfo, file, line, source_text + " invoked");
- return function_mocker_->AddNewExpectation(
- file, line, source_text, matchers_);
+ return function_mocker_->AddNewExpectation(file, line, source_text,
+ matchers_);
}
// This operator overload is used to swallow the superfluous parameter list
@@ -1317,9 +1312,7 @@
class ReferenceOrValueWrapper {
public:
// Constructs a wrapper from the given value/reference.
- explicit ReferenceOrValueWrapper(T value)
- : value_(std::move(value)) {
- }
+ explicit ReferenceOrValueWrapper(T value) : value_(std::move(value)) {}
// Unwraps and returns the underlying value/reference, exactly as
// originally passed. The behavior of calling this more than once on
@@ -1330,9 +1323,7 @@
// Always returns a const reference (more precisely,
// const std::add_lvalue_reference<T>::type). The behavior of calling this
// after calling Unwrap on the same object is unspecified.
- const T& Peek() const {
- return value_;
- }
+ const T& Peek() const { return value_; }
private:
T value_;
@@ -1346,8 +1337,7 @@
// Workaround for debatable pass-by-reference lint warning (c-library-team
// policy precludes NOLINT in this context)
typedef T& reference;
- explicit ReferenceOrValueWrapper(reference ref)
- : value_ptr_(&ref) {}
+ explicit ReferenceOrValueWrapper(reference ref) : value_ptr_(&ref) {}
T& Unwrap() { return *value_ptr_; }
const T& Peek() const { return *value_ptr_; }
@@ -1355,102 +1345,27 @@
T* value_ptr_;
};
-// C++ treats the void type specially. For example, you cannot define
-// a void-typed variable or pass a void value to a function.
-// ActionResultHolder<T> holds a value of type T, where T must be a
-// copyable type or void (T doesn't need to be default-constructable).
-// It hides the syntactic difference between void and other types, and
-// is used to unify the code for invoking both void-returning and
-// non-void-returning mock functions.
-
-// Untyped base class for ActionResultHolder<T>.
-class UntypedActionResultHolderBase {
- public:
- virtual ~UntypedActionResultHolderBase() {}
-
- // Prints the held value as an action's result to os.
- virtual void PrintAsActionResult(::std::ostream* os) const = 0;
-};
-
-// This generic definition is used when T is not void.
+// Prints the held value as an action's result to os.
template <typename T>
-class ActionResultHolder : public UntypedActionResultHolderBase {
+void PrintAsActionResult(const T& result, std::ostream& os) {
+ os << "\n Returns: ";
+ // T may be a reference type, so we don't use UniversalPrint().
+ UniversalPrinter<T>::Print(result, &os);
+}
+
+// Reports an uninteresting call (whose description is in msg) in the
+// manner specified by 'reaction'.
+GTEST_API_ void ReportUninterestingCall(CallReaction reaction,
+ const std::string& msg);
+
+// A generic RAII type that runs a user-provided function in its destructor.
+class Cleanup final {
public:
- // Returns the held value. Must not be called more than once.
- T Unwrap() {
- return result_.Unwrap();
- }
-
- // Prints the held value as an action's result to os.
- void PrintAsActionResult(::std::ostream* os) const override {
- *os << "\n Returns: ";
- // T may be a reference type, so we don't use UniversalPrint().
- UniversalPrinter<T>::Print(result_.Peek(), os);
- }
-
- // Performs the given mock function's default action and returns the
- // result in a new-ed ActionResultHolder.
- template <typename F>
- static ActionResultHolder* PerformDefaultAction(
- const FunctionMocker<F>* func_mocker,
- typename Function<F>::ArgumentTuple&& args,
- const std::string& call_description) {
- return new ActionResultHolder(Wrapper(func_mocker->PerformDefaultAction(
- std::move(args), call_description)));
- }
-
- // Performs the given action and returns the result in a new-ed
- // ActionResultHolder.
- template <typename F>
- static ActionResultHolder* PerformAction(
- const Action<F>& action, typename Function<F>::ArgumentTuple&& args) {
- return new ActionResultHolder(
- Wrapper(action.Perform(std::move(args))));
- }
+ explicit Cleanup(std::function<void()> f) : f_(std::move(f)) {}
+ ~Cleanup() { f_(); }
private:
- typedef ReferenceOrValueWrapper<T> Wrapper;
-
- explicit ActionResultHolder(Wrapper result)
- : result_(std::move(result)) {
- }
-
- Wrapper result_;
-
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ActionResultHolder);
-};
-
-// Specialization for T = void.
-template <>
-class ActionResultHolder<void> : public UntypedActionResultHolderBase {
- public:
- void Unwrap() { }
-
- void PrintAsActionResult(::std::ostream* /* os */) const override {}
-
- // Performs the given mock function's default action and returns ownership
- // of an empty ActionResultHolder*.
- template <typename F>
- static ActionResultHolder* PerformDefaultAction(
- const FunctionMocker<F>* func_mocker,
- typename Function<F>::ArgumentTuple&& args,
- const std::string& call_description) {
- func_mocker->PerformDefaultAction(std::move(args), call_description);
- return new ActionResultHolder;
- }
-
- // Performs the given action and returns ownership of an empty
- // ActionResultHolder*.
- template <typename F>
- static ActionResultHolder* PerformAction(
- const Action<F>& action, typename Function<F>::ArgumentTuple&& args) {
- action.Perform(std::move(args));
- return new ActionResultHolder;
- }
-
- private:
- ActionResultHolder() {}
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ActionResultHolder);
+ std::function<void()> f_;
};
template <typename F>
@@ -1495,14 +1410,12 @@
// Returns the ON_CALL spec that matches this mock function with the
// given arguments; returns NULL if no matching ON_CALL is found.
// L = *
- const OnCallSpec<F>* FindOnCallSpec(
- const ArgumentTuple& args) const {
- for (UntypedOnCallSpecs::const_reverse_iterator it
- = untyped_on_call_specs_.rbegin();
+ const OnCallSpec<F>* FindOnCallSpec(const ArgumentTuple& args) const {
+ for (UntypedOnCallSpecs::const_reverse_iterator it =
+ untyped_on_call_specs_.rbegin();
it != untyped_on_call_specs_.rend(); ++it) {
const OnCallSpec<F>* spec = static_cast<const OnCallSpec<F>*>(*it);
- if (spec->Matches(args))
- return spec;
+ if (spec->Matches(args)) return spec;
}
return nullptr;
@@ -1510,15 +1423,14 @@
// Performs the default action of this mock function on the given
// arguments and returns the result. Asserts (or throws if
- // exceptions are enabled) with a helpful call descrption if there
+ // exceptions are enabled) with a helpful call description if there
// is no valid return value. This method doesn't depend on the
// mutable state of this object, and thus can be called concurrently
// without locking.
// L = *
Result PerformDefaultAction(ArgumentTuple&& args,
const std::string& call_description) const {
- const OnCallSpec<F>* const spec =
- this->FindOnCallSpec(args);
+ const OnCallSpec<F>* const spec = this->FindOnCallSpec(args);
if (spec != nullptr) {
return spec->GetAction().Perform(std::move(args));
}
@@ -1536,32 +1448,6 @@
return DefaultValue<Result>::Get();
}
- // Performs the default action with the given arguments and returns
- // the action's result. The call description string will be used in
- // the error message to describe the call in the case the default
- // action fails. The caller is responsible for deleting the result.
- // L = *
- UntypedActionResultHolderBase* UntypedPerformDefaultAction(
- void* untyped_args, // must point to an ArgumentTuple
- const std::string& call_description) const override {
- ArgumentTuple* args = static_cast<ArgumentTuple*>(untyped_args);
- return ResultHolder::PerformDefaultAction(this, std::move(*args),
- call_description);
- }
-
- // Performs the given action with the given arguments and returns
- // the action's result. The caller is responsible for deleting the
- // result.
- // L = *
- UntypedActionResultHolderBase* UntypedPerformAction(
- const void* untyped_action, void* untyped_args) const override {
- // Make a copy of the action before performing it, in case the
- // action deletes the mock object (and thus deletes itself).
- const Action<F> action = *static_cast<const Action<F>*>(untyped_action);
- ArgumentTuple* args = static_cast<ArgumentTuple*>(untyped_args);
- return ResultHolder::PerformAction(action, std::move(*args));
- }
-
// Implements UntypedFunctionMockerBase::ClearDefaultActionsLocked():
// clears the ON_CALL()s set on this mock function.
void ClearDefaultActionsLocked() override
@@ -1579,8 +1465,7 @@
untyped_on_call_specs_.swap(specs_to_delete);
g_gmock_mutex.Unlock();
- for (UntypedOnCallSpecs::const_iterator it =
- specs_to_delete.begin();
+ for (UntypedOnCallSpecs::const_iterator it = specs_to_delete.begin();
it != specs_to_delete.end(); ++it) {
delete static_cast<const OnCallSpec<F>*>(*it);
}
@@ -1594,10 +1479,7 @@
// arguments. This function can be safely called from multiple
// threads concurrently.
Result Invoke(Args... args) GTEST_LOCK_EXCLUDED_(g_gmock_mutex) {
- ArgumentTuple tuple(std::forward<Args>(args)...);
- std::unique_ptr<ResultHolder> holder(DownCast_<ResultHolder*>(
- this->UntypedInvokeWith(static_cast<void*>(&tuple))));
- return holder->Unwrap();
+ return InvokeWith(ArgumentTuple(std::forward<Args>(args)...));
}
MockSpec<F> With(Matcher<Args>... m) {
@@ -1608,13 +1490,10 @@
template <typename Function>
friend class MockSpec;
- typedef ActionResultHolder<Result> ResultHolder;
-
// Adds and returns a default action spec for this mock function.
- OnCallSpec<F>& AddNewOnCallSpec(
- const char* file, int line,
- const ArgumentMatcherTuple& m)
- GTEST_LOCK_EXCLUDED_(g_gmock_mutex) {
+ OnCallSpec<F>& AddNewOnCallSpec(const char* file, int line,
+ const ArgumentMatcherTuple& m)
+ GTEST_LOCK_EXCLUDED_(g_gmock_mutex) {
Mock::RegisterUseByOnCallOrExpectCall(MockObject(), file, line);
OnCallSpec<F>* const on_call_spec = new OnCallSpec<F>(file, line, m);
untyped_on_call_specs_.push_back(on_call_spec);
@@ -1644,7 +1523,8 @@
}
private:
- template <typename Func> friend class TypedExpectation;
+ template <typename Func>
+ friend class TypedExpectation;
// Some utilities needed for implementing UntypedInvokeWith().
@@ -1728,9 +1608,8 @@
// Returns the expectation that matches the arguments, or NULL if no
// expectation matches them.
- TypedExpectation<F>* FindMatchingExpectationLocked(
- const ArgumentTuple& args) const
- GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
+ TypedExpectation<F>* FindMatchingExpectationLocked(const ArgumentTuple& args)
+ const GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
g_gmock_mutex.AssertHeld();
// See the definition of untyped_expectations_ for why access to
// it is unprotected here.
@@ -1747,11 +1626,10 @@
}
// Returns a message that the arguments don't match any expectation.
- void FormatUnexpectedCallMessageLocked(
- const ArgumentTuple& args,
- ::std::ostream* os,
- ::std::ostream* why) const
- GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
+ void FormatUnexpectedCallMessageLocked(const ArgumentTuple& args,
+ ::std::ostream* os,
+ ::std::ostream* why) const
+ GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
g_gmock_mutex.AssertHeld();
*os << "\nUnexpected mock function call - ";
DescribeDefaultActionTo(args, os);
@@ -1760,15 +1638,14 @@
// Prints a list of expectations that have been tried against the
// current mock function call.
- void PrintTriedExpectationsLocked(
- const ArgumentTuple& args,
- ::std::ostream* why) const
- GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
+ void PrintTriedExpectationsLocked(const ArgumentTuple& args,
+ ::std::ostream* why) const
+ GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
g_gmock_mutex.AssertHeld();
const size_t count = untyped_expectations_.size();
*why << "Google Mock tried the following " << count << " "
- << (count == 1 ? "expectation, but it didn't match" :
- "expectations, but none matched")
+ << (count == 1 ? "expectation, but it didn't match"
+ : "expectations, but none matched")
<< ":\n";
for (size_t i = 0; i < count; i++) {
TypedExpectation<F>* const expectation =
@@ -1783,11 +1660,177 @@
expectation->DescribeCallCountTo(why);
}
}
+
+ // Performs the given action (or the default if it's null) with the given
+ // arguments and returns the action's result.
+ // L = *
+ R PerformAction(const void* untyped_action, ArgumentTuple&& args,
+ const std::string& call_description) const {
+ if (untyped_action == nullptr) {
+ return PerformDefaultAction(std::move(args), call_description);
+ }
+
+ // Make a copy of the action before performing it, in case the
+ // action deletes the mock object (and thus deletes itself).
+ const Action<F> action = *static_cast<const Action<F>*>(untyped_action);
+ return action.Perform(std::move(args));
+ }
+
+ // Is it possible to store an object of the supplied type in a local variable
+ // for the sake of printing it, then return it on to the caller?
+ template <typename T>
+ using can_print_result = internal::conjunction<
+ // void can't be stored as an object (and we also don't need to print it).
+ internal::negation<std::is_void<T>>,
+ // Non-moveable types can't be returned on to the user, so there's no way
+ // for us to intercept and print them.
+ std::is_move_constructible<T>>;
+
+ // Perform the supplied action, printing the result to os.
+ template <typename T = R,
+ typename std::enable_if<can_print_result<T>::value, int>::type = 0>
+ R PerformActionAndPrintResult(const void* const untyped_action,
+ ArgumentTuple&& args,
+ const std::string& call_description,
+ std::ostream& os) {
+ R result = PerformAction(untyped_action, std::move(args), call_description);
+
+ PrintAsActionResult(result, os);
+ return std::forward<R>(result);
+ }
+
+ // An overload for when it's not possible to print the result. In this case we
+ // simply perform the action.
+ template <typename T = R,
+ typename std::enable_if<
+ internal::negation<can_print_result<T>>::value, int>::type = 0>
+ R PerformActionAndPrintResult(const void* const untyped_action,
+ ArgumentTuple&& args,
+ const std::string& call_description,
+ std::ostream&) {
+ return PerformAction(untyped_action, std::move(args), call_description);
+ }
+
+ // Returns the result of invoking this mock function with the given
+ // arguments. This function can be safely called from multiple
+ // threads concurrently.
+ R InvokeWith(ArgumentTuple&& args) GTEST_LOCK_EXCLUDED_(g_gmock_mutex);
}; // class FunctionMocker
-// Reports an uninteresting call (whose description is in msg) in the
-// manner specified by 'reaction'.
-void ReportUninterestingCall(CallReaction reaction, const std::string& msg);
+// Calculates the result of invoking this mock function with the given
+// arguments, prints it, and returns it.
+template <typename R, typename... Args>
+R FunctionMocker<R(Args...)>::InvokeWith(ArgumentTuple&& args)
+ GTEST_LOCK_EXCLUDED_(g_gmock_mutex) {
+ // See the definition of untyped_expectations_ for why access to it
+ // is unprotected here.
+ if (untyped_expectations_.size() == 0) {
+ // No expectation is set on this mock method - we have an
+ // uninteresting call.
+
+ // We must get Google Mock's reaction on uninteresting calls
+ // made on this mock object BEFORE performing the action,
+ // because the action may DELETE the mock object and make the
+ // following expression meaningless.
+ const CallReaction reaction =
+ Mock::GetReactionOnUninterestingCalls(MockObject());
+
+ // True if and only if we need to print this call's arguments and return
+ // value. This definition must be kept in sync with
+ // the behavior of ReportUninterestingCall().
+ const bool need_to_report_uninteresting_call =
+ // If the user allows this uninteresting call, we print it
+ // only when they want informational messages.
+ reaction == kAllow ? LogIsVisible(kInfo) :
+ // If the user wants this to be a warning, we print
+ // it only when they want to see warnings.
+ reaction == kWarn
+ ? LogIsVisible(kWarning)
+ :
+ // Otherwise, the user wants this to be an error, and we
+ // should always print detailed information in the error.
+ true;
+
+ if (!need_to_report_uninteresting_call) {
+ // Perform the action without printing the call information.
+ return this->PerformDefaultAction(
+ std::move(args), "Function call: " + std::string(Name()));
+ }
+
+ // Warns about the uninteresting call.
+ ::std::stringstream ss;
+ this->UntypedDescribeUninterestingCall(&args, &ss);
+
+ // Perform the action, print the result, and then report the uninteresting
+ // call.
+ //
+ // We use RAII to do the latter in case R is void or a non-moveable type. In
+ // either case we can't assign it to a local variable.
+ const Cleanup report_uninteresting_call(
+ [&] { ReportUninterestingCall(reaction, ss.str()); });
+
+ return PerformActionAndPrintResult(nullptr, std::move(args), ss.str(), ss);
+ }
+
+ bool is_excessive = false;
+ ::std::stringstream ss;
+ ::std::stringstream why;
+ ::std::stringstream loc;
+ const void* untyped_action = nullptr;
+
+ // The UntypedFindMatchingExpectation() function acquires and
+ // releases g_gmock_mutex.
+
+ const ExpectationBase* const untyped_expectation =
+ this->UntypedFindMatchingExpectation(&args, &untyped_action,
+ &is_excessive, &ss, &why);
+ const bool found = untyped_expectation != nullptr;
+
+ // True if and only if we need to print the call's arguments
+ // and return value.
+ // This definition must be kept in sync with the uses of Expect()
+ // and Log() in this function.
+ const bool need_to_report_call =
+ !found || is_excessive || LogIsVisible(kInfo);
+ if (!need_to_report_call) {
+ // Perform the action without printing the call information.
+ return PerformAction(untyped_action, std::move(args), "");
+ }
+
+ ss << " Function call: " << Name();
+ this->UntypedPrintArgs(&args, &ss);
+
+ // In case the action deletes a piece of the expectation, we
+ // generate the message beforehand.
+ if (found && !is_excessive) {
+ untyped_expectation->DescribeLocationTo(&loc);
+ }
+
+ // Perform the action, print the result, and then fail or log in whatever way
+ // is appropriate.
+ //
+ // We use RAII to do the latter in case R is void or a non-moveable type. In
+ // either case we can't assign it to a local variable.
+ const Cleanup handle_failures([&] {
+ ss << "\n" << why.str();
+
+ if (!found) {
+ // No expectation matches this call - reports a failure.
+ Expect(false, nullptr, -1, ss.str());
+ } else if (is_excessive) {
+ // We had an upper-bound violation and the failure message is in ss.
+ Expect(false, untyped_expectation->file(), untyped_expectation->line(),
+ ss.str());
+ } else {
+ // We had an expected call and the matching expectation is
+ // described in ss.
+ Log(kInfo, loc.str() + ss.str(), 2);
+ }
+ });
+
+ return PerformActionAndPrintResult(untyped_action, std::move(args), ss.str(),
+ ss);
+}
} // namespace internal
@@ -1952,7 +1995,9 @@
// // Expects a call to const MockFoo::Bar().
// EXPECT_CALL(Const(foo), Bar());
template <typename T>
-inline const T& Const(const T& x) { return x; }
+inline const T& Const(const T& x) {
+ return x;
+}
// Constructs an Expectation object that references and co-owns exp.
inline Expectation::Expectation(internal::ExpectationBase& exp) // NOLINT
diff --git a/third_party/googletest/src/googlemock/include/gmock/gmock.h b/third_party/googletest/src/googlemock/include/gmock/gmock.h
index 12469bc..568c8c7 100644
--- a/third_party/googletest/src/googlemock/include/gmock/gmock.h
+++ b/third_party/googletest/src/googlemock/include/gmock/gmock.h
@@ -27,13 +27,10 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Google Mock - a framework for writing C++ mock classes.
//
// This is the main header file a user should include.
-// GOOGLETEST_CM0002 DO NOT DELETE
-
#ifndef GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_H_
#define GOOGLEMOCK_INCLUDE_GMOCK_GMOCK_H_
@@ -64,14 +61,15 @@
#include "gmock/gmock-more-matchers.h"
#include "gmock/gmock-nice-strict.h"
#include "gmock/internal/gmock-internal-utils.h"
-
-namespace testing {
+#include "gmock/internal/gmock-port.h"
// Declares Google Mock flags that we want a user to use programmatically.
GMOCK_DECLARE_bool_(catch_leaked_mocks);
GMOCK_DECLARE_string_(verbose);
GMOCK_DECLARE_int32_(default_mock_behavior);
+namespace testing {
+
// Initializes Google Mock. This must be called before running the
// tests. In particular, it parses the command line for the flags
// that Google Mock recognizes. Whenever a Google Mock flag is seen,
diff --git a/third_party/googletest/src/googlemock/include/gmock/internal/custom/README.md b/third_party/googletest/src/googlemock/include/gmock/internal/custom/README.md
index f6c93f6..9c4874f 100644
--- a/third_party/googletest/src/googlemock/include/gmock/internal/custom/README.md
+++ b/third_party/googletest/src/googlemock/include/gmock/internal/custom/README.md
@@ -14,3 +14,5 @@
* `GMOCK_DEFINE_bool_(name, default_val, doc)`
* `GMOCK_DEFINE_int32_(name, default_val, doc)`
* `GMOCK_DEFINE_string_(name, default_val, doc)`
+* `GMOCK_FLAG_GET(flag_name)`
+* `GMOCK_FLAG_SET(flag_name, value)`
diff --git a/third_party/googletest/src/googlemock/include/gmock/internal/custom/gmock-generated-actions.h b/third_party/googletest/src/googlemock/include/gmock/internal/custom/gmock-generated-actions.h
index 63f8999..bbcad31 100644
--- a/third_party/googletest/src/googlemock/include/gmock/internal/custom/gmock-generated-actions.h
+++ b/third_party/googletest/src/googlemock/include/gmock/internal/custom/gmock-generated-actions.h
@@ -1,4 +1,5 @@
-// GOOGLETEST_CM0002 DO NOT DELETE
+// IWYU pragma: private, include "gmock/gmock.h"
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLEMOCK_INCLUDE_GMOCK_INTERNAL_CUSTOM_GMOCK_GENERATED_ACTIONS_H_
#define GOOGLEMOCK_INCLUDE_GMOCK_INTERNAL_CUSTOM_GMOCK_GENERATED_ACTIONS_H_
diff --git a/third_party/googletest/src/googlemock/include/gmock/internal/custom/gmock-matchers.h b/third_party/googletest/src/googlemock/include/gmock/internal/custom/gmock-matchers.h
index 6384294..bb7dcba 100644
--- a/third_party/googletest/src/googlemock/include/gmock/internal/custom/gmock-matchers.h
+++ b/third_party/googletest/src/googlemock/include/gmock/internal/custom/gmock-matchers.h
@@ -26,10 +26,11 @@
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
+
// Injection point for custom user configurations. See README for details
-//
-// GOOGLETEST_CM0002 DO NOT DELETE
+
+// IWYU pragma: private, include "gmock/gmock.h"
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLEMOCK_INCLUDE_GMOCK_INTERNAL_CUSTOM_GMOCK_MATCHERS_H_
#define GOOGLEMOCK_INCLUDE_GMOCK_INTERNAL_CUSTOM_GMOCK_MATCHERS_H_
diff --git a/third_party/googletest/src/googlemock/include/gmock/internal/custom/gmock-port.h b/third_party/googletest/src/googlemock/include/gmock/internal/custom/gmock-port.h
index 1437869..f055f75 100644
--- a/third_party/googletest/src/googlemock/include/gmock/internal/custom/gmock-port.h
+++ b/third_party/googletest/src/googlemock/include/gmock/internal/custom/gmock-port.h
@@ -26,12 +26,13 @@
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
+
// Injection point for custom user configurations. See README for details
//
// ** Custom implementation starts here **
-// GOOGLETEST_CM0002 DO NOT DELETE
+// IWYU pragma: private, include "gmock/gmock.h"
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLEMOCK_INCLUDE_GMOCK_INTERNAL_CUSTOM_GMOCK_PORT_H_
#define GOOGLEMOCK_INCLUDE_GMOCK_INTERNAL_CUSTOM_GMOCK_PORT_H_
diff --git a/third_party/googletest/src/googlemock/include/gmock/internal/gmock-internal-utils.h b/third_party/googletest/src/googlemock/include/gmock/internal/gmock-internal-utils.h
index 317544a..b1343fd 100644
--- a/third_party/googletest/src/googlemock/include/gmock/internal/gmock-internal-utils.h
+++ b/third_party/googletest/src/googlemock/include/gmock/internal/gmock-internal-utils.h
@@ -27,22 +27,25 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Google Mock - a framework for writing C++ mock classes.
//
// This file defines some utilities useful for implementing Google
// Mock. They are subject to change without notice, so please DO NOT
// USE THEM IN USER CODE.
-// GOOGLETEST_CM0002 DO NOT DELETE
+// IWYU pragma: private, include "gmock/gmock.h"
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLEMOCK_INCLUDE_GMOCK_INTERNAL_GMOCK_INTERNAL_UTILS_H_
#define GOOGLEMOCK_INCLUDE_GMOCK_INTERNAL_GMOCK_INTERNAL_UTILS_H_
#include <stdio.h>
+
#include <ostream> // NOLINT
#include <string>
#include <type_traits>
+#include <vector>
+
#include "gmock/internal/gmock-port.h"
#include "gtest/gtest.h"
@@ -56,14 +59,15 @@
// Silence MSVC C4100 (unreferenced formal parameter) and
// C4805('==': unsafe mix of type 'const int' and type 'const bool')
#ifdef _MSC_VER
-# pragma warning(push)
-# pragma warning(disable:4100)
-# pragma warning(disable:4805)
+#pragma warning(push)
+#pragma warning(disable : 4100)
+#pragma warning(disable : 4805)
#endif
// Joins a vector of strings as if they are fields of a tuple; returns
// the joined string.
-GTEST_API_ std::string JoinAsTuple(const Strings& fields);
+GTEST_API_ std::string JoinAsKeyValueTuple(
+ const std::vector<const char*>& names, const Strings& values);
// Converts an identifier name to a space-separated list of lower-case
// words. Each maximum substring of the form [A-Za-z][a-z]*|\d+ is
@@ -78,9 +82,18 @@
inline const typename Pointer::element_type* GetRawPointer(const Pointer& p) {
return p.get();
}
+// This overload version is for std::reference_wrapper, which does not work with
+// the overload above, as it does not have an `element_type`.
+template <typename Element>
+inline const Element* GetRawPointer(const std::reference_wrapper<Element>& r) {
+ return &r.get();
+}
+
// This overloaded version is for the raw pointer case.
template <typename Element>
-inline Element* GetRawPointer(Element* p) { return p; }
+inline Element* GetRawPointer(Element* p) {
+ return p;
+}
// MSVC treats wchar_t as a native type usually, but treats it as the
// same as unsigned short when the compiler option /Zc:wchar_t- is
@@ -89,7 +102,7 @@
#if defined(_MSC_VER) && !defined(_NATIVE_WCHAR_T_DEFINED)
// wchar_t is a typedef.
#else
-# define GMOCK_WCHAR_T_IS_NATIVE_ 1
+#define GMOCK_WCHAR_T_IS_NATIVE_ 1
#endif
// In what follows, we use the term "kind" to indicate whether a type
@@ -97,18 +110,20 @@
// or none of them. This categorization is useful for determining
// when a matcher argument type can be safely converted to another
// type in the implementation of SafeMatcherCast.
-enum TypeKind {
- kBool, kInteger, kFloatingPoint, kOther
-};
+enum TypeKind { kBool, kInteger, kFloatingPoint, kOther };
// KindOf<T>::value is the kind of type T.
-template <typename T> struct KindOf {
+template <typename T>
+struct KindOf {
enum { value = kOther }; // The default kind.
};
// This macro declares that the kind of 'type' is 'kind'.
#define GMOCK_DECLARE_KIND_(type, kind) \
- template <> struct KindOf<type> { enum { value = kind }; }
+ template <> \
+ struct KindOf<type> { \
+ enum { value = kind }; \
+ }
GMOCK_DECLARE_KIND_(bool, kBool);
@@ -116,13 +131,13 @@
GMOCK_DECLARE_KIND_(char, kInteger);
GMOCK_DECLARE_KIND_(signed char, kInteger);
GMOCK_DECLARE_KIND_(unsigned char, kInteger);
-GMOCK_DECLARE_KIND_(short, kInteger); // NOLINT
+GMOCK_DECLARE_KIND_(short, kInteger); // NOLINT
GMOCK_DECLARE_KIND_(unsigned short, kInteger); // NOLINT
GMOCK_DECLARE_KIND_(int, kInteger);
GMOCK_DECLARE_KIND_(unsigned int, kInteger);
-GMOCK_DECLARE_KIND_(long, kInteger); // NOLINT
-GMOCK_DECLARE_KIND_(unsigned long, kInteger); // NOLINT
-GMOCK_DECLARE_KIND_(long long, kInteger); // NOLINT
+GMOCK_DECLARE_KIND_(long, kInteger); // NOLINT
+GMOCK_DECLARE_KIND_(unsigned long, kInteger); // NOLINT
+GMOCK_DECLARE_KIND_(long long, kInteger); // NOLINT
GMOCK_DECLARE_KIND_(unsigned long long, kInteger); // NOLINT
#if GMOCK_WCHAR_T_IS_NATIVE_
@@ -137,7 +152,7 @@
#undef GMOCK_DECLARE_KIND_
// Evaluates to the kind of 'type'.
-#define GMOCK_KIND_OF_(type) \
+#define GMOCK_KIND_OF_(type) \
static_cast< ::testing::internal::TypeKind>( \
::testing::internal::KindOf<type>::value)
@@ -193,9 +208,7 @@
class FailureReporterInterface {
public:
// The type of a failure (either non-fatal or fatal).
- enum FailureType {
- kNonfatal, kFatal
- };
+ enum FailureType { kNonfatal, kFatal };
virtual ~FailureReporterInterface() {}
@@ -215,8 +228,8 @@
inline void Assert(bool condition, const char* file, int line,
const std::string& msg) {
if (!condition) {
- GetFailureReporter()->ReportFailure(FailureReporterInterface::kFatal,
- file, line, msg);
+ GetFailureReporter()->ReportFailure(FailureReporterInterface::kFatal, file,
+ line, msg);
}
}
inline void Assert(bool condition, const char* file, int line) {
@@ -237,10 +250,7 @@
}
// Severity level of a log.
-enum LogSeverity {
- kInfo = 0,
- kWarning = 1
-};
+enum LogSeverity { kInfo = 0, kWarning = 1 };
// Valid values for the --gmock_verbose flag.
@@ -281,10 +291,10 @@
GTEST_API_ WithoutMatchers GetWithoutMatchers();
// Disable MSVC warnings for infinite recursion, since in this case the
-// the recursion is unreachable.
+// recursion is unreachable.
#ifdef _MSC_VER
-# pragma warning(push)
-# pragma warning(disable:4717)
+#pragma warning(push)
+#pragma warning(disable : 4717)
#endif
// Invalid<T>() is usable as an expression of type T, but will terminate
@@ -295,14 +305,17 @@
template <typename T>
inline T Invalid() {
Assert(false, "", -1, "Internal error: attempt to return invalid value");
- // This statement is unreachable, and would never terminate even if it
- // could be reached. It is provided only to placate compiler warnings
- // about missing return statements.
+#if defined(__GNUC__) || defined(__clang__)
+ __builtin_unreachable();
+#elif defined(_MSC_VER)
+ __assume(0);
+#else
return Invalid<T>();
+#endif
}
#ifdef _MSC_VER
-# pragma warning(pop)
+#pragma warning(pop)
#endif
// Given a raw type (i.e. having no top-level reference or const
@@ -381,7 +394,8 @@
// The following specialization prevents the user from instantiating
// StlContainer with a reference type.
-template <typename T> class StlContainerView<T&>;
+template <typename T>
+class StlContainerView<T&>;
// A type transform to remove constness from the first part of a pair.
// Pairs like that are used as the value_type of associative containers,
@@ -402,17 +416,18 @@
GTEST_API_ void IllegalDoDefault(const char* file, int line);
template <typename F, typename Tuple, size_t... Idx>
-auto ApplyImpl(F&& f, Tuple&& args, IndexSequence<Idx...>) -> decltype(
- std::forward<F>(f)(std::get<Idx>(std::forward<Tuple>(args))...)) {
+auto ApplyImpl(F&& f, Tuple&& args, IndexSequence<Idx...>)
+ -> decltype(std::forward<F>(f)(
+ std::get<Idx>(std::forward<Tuple>(args))...)) {
return std::forward<F>(f)(std::get<Idx>(std::forward<Tuple>(args))...);
}
// Apply the function to a tuple of arguments.
template <typename F, typename Tuple>
-auto Apply(F&& f, Tuple&& args) -> decltype(
- ApplyImpl(std::forward<F>(f), std::forward<Tuple>(args),
- MakeIndexSequence<std::tuple_size<
- typename std::remove_reference<Tuple>::type>::value>())) {
+auto Apply(F&& f, Tuple&& args) -> decltype(ApplyImpl(
+ std::forward<F>(f), std::forward<Tuple>(args),
+ MakeIndexSequence<std::tuple_size<
+ typename std::remove_reference<Tuple>::type>::value>())) {
return ApplyImpl(std::forward<F>(f), std::forward<Tuple>(args),
MakeIndexSequence<std::tuple_size<
typename std::remove_reference<Tuple>::type>::value>());
@@ -449,8 +464,10 @@
template <typename R, typename... Args>
constexpr size_t Function<R(Args...)>::ArgumentCount;
+bool Base64Unescape(const std::string& encoded, std::string* decoded);
+
#ifdef _MSC_VER
-# pragma warning(pop)
+#pragma warning(pop)
#endif
} // namespace internal
diff --git a/third_party/googletest/src/googlemock/include/gmock/internal/gmock-port.h b/third_party/googletest/src/googlemock/include/gmock/internal/gmock-port.h
index 367a44d..bc18a25 100644
--- a/third_party/googletest/src/googlemock/include/gmock/internal/gmock-port.h
+++ b/third_party/googletest/src/googlemock/include/gmock/internal/gmock-port.h
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
// Low-level types and utilities for porting Google Mock to various
// platforms. All macros ending with _ and symbols defined in an
// internal namespace are subject to change without notice. Code
@@ -35,7 +34,8 @@
// end with _ are part of Google Mock's public API and can be used by
// code outside Google Mock.
-// GOOGLETEST_CM0002 DO NOT DELETE
+// IWYU pragma: private, include "gmock/gmock.h"
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLEMOCK_INCLUDE_GMOCK_INTERNAL_GMOCK_PORT_H_
#define GOOGLEMOCK_INCLUDE_GMOCK_INTERNAL_GMOCK_PORT_H_
@@ -53,35 +53,87 @@
// here, as Google Mock depends on Google Test. Only add a utility
// here if it's truly specific to Google Mock.
-#include "gtest/internal/gtest-port.h"
#include "gmock/internal/custom/gmock-port.h"
+#include "gtest/internal/gtest-port.h"
+
+#if GTEST_HAS_ABSL
+#include "absl/flags/declare.h"
+#include "absl/flags/flag.h"
+#endif
// For MS Visual C++, check the compiler version. At least VS 2015 is
// required to compile Google Mock.
#if defined(_MSC_VER) && _MSC_VER < 1900
-# error "At least Visual C++ 2015 (14.0) is required to compile Google Mock."
+#error "At least Visual C++ 2015 (14.0) is required to compile Google Mock."
#endif
// Macro for referencing flags. This is public as we want the user to
// use this syntax to reference Google Mock flags.
+#define GMOCK_FLAG_NAME_(name) gmock_##name
#define GMOCK_FLAG(name) FLAGS_gmock_##name
-#if !defined(GMOCK_DECLARE_bool_)
-
-// Macros for declaring flags.
-# define GMOCK_DECLARE_bool_(name) extern GTEST_API_ bool GMOCK_FLAG(name)
-# define GMOCK_DECLARE_int32_(name) extern GTEST_API_ int32_t GMOCK_FLAG(name)
-# define GMOCK_DECLARE_string_(name) \
- extern GTEST_API_ ::std::string GMOCK_FLAG(name)
+// Pick a command line flags implementation.
+#if GTEST_HAS_ABSL
// Macros for defining flags.
-# define GMOCK_DEFINE_bool_(name, default_val, doc) \
- GTEST_API_ bool GMOCK_FLAG(name) = (default_val)
-# define GMOCK_DEFINE_int32_(name, default_val, doc) \
- GTEST_API_ int32_t GMOCK_FLAG(name) = (default_val)
-# define GMOCK_DEFINE_string_(name, default_val, doc) \
- GTEST_API_ ::std::string GMOCK_FLAG(name) = (default_val)
+#define GMOCK_DEFINE_bool_(name, default_val, doc) \
+ ABSL_FLAG(bool, GMOCK_FLAG_NAME_(name), default_val, doc)
+#define GMOCK_DEFINE_int32_(name, default_val, doc) \
+ ABSL_FLAG(int32_t, GMOCK_FLAG_NAME_(name), default_val, doc)
+#define GMOCK_DEFINE_string_(name, default_val, doc) \
+ ABSL_FLAG(std::string, GMOCK_FLAG_NAME_(name), default_val, doc)
-#endif // !defined(GMOCK_DECLARE_bool_)
+// Macros for declaring flags.
+#define GMOCK_DECLARE_bool_(name) \
+ ABSL_DECLARE_FLAG(bool, GMOCK_FLAG_NAME_(name))
+#define GMOCK_DECLARE_int32_(name) \
+ ABSL_DECLARE_FLAG(int32_t, GMOCK_FLAG_NAME_(name))
+#define GMOCK_DECLARE_string_(name) \
+ ABSL_DECLARE_FLAG(std::string, GMOCK_FLAG_NAME_(name))
+
+#define GMOCK_FLAG_GET(name) ::absl::GetFlag(GMOCK_FLAG(name))
+#define GMOCK_FLAG_SET(name, value) \
+ (void)(::absl::SetFlag(&GMOCK_FLAG(name), value))
+
+#else // GTEST_HAS_ABSL
+
+// Macros for defining flags.
+#define GMOCK_DEFINE_bool_(name, default_val, doc) \
+ namespace testing { \
+ GTEST_API_ bool GMOCK_FLAG(name) = (default_val); \
+ } \
+ static_assert(true, "no-op to require trailing semicolon")
+#define GMOCK_DEFINE_int32_(name, default_val, doc) \
+ namespace testing { \
+ GTEST_API_ int32_t GMOCK_FLAG(name) = (default_val); \
+ } \
+ static_assert(true, "no-op to require trailing semicolon")
+#define GMOCK_DEFINE_string_(name, default_val, doc) \
+ namespace testing { \
+ GTEST_API_ ::std::string GMOCK_FLAG(name) = (default_val); \
+ } \
+ static_assert(true, "no-op to require trailing semicolon")
+
+// Macros for declaring flags.
+#define GMOCK_DECLARE_bool_(name) \
+ namespace testing { \
+ GTEST_API_ extern bool GMOCK_FLAG(name); \
+ } \
+ static_assert(true, "no-op to require trailing semicolon")
+#define GMOCK_DECLARE_int32_(name) \
+ namespace testing { \
+ GTEST_API_ extern int32_t GMOCK_FLAG(name); \
+ } \
+ static_assert(true, "no-op to require trailing semicolon")
+#define GMOCK_DECLARE_string_(name) \
+ namespace testing { \
+ GTEST_API_ extern ::std::string GMOCK_FLAG(name); \
+ } \
+ static_assert(true, "no-op to require trailing semicolon")
+
+#define GMOCK_FLAG_GET(name) ::testing::GMOCK_FLAG(name)
+#define GMOCK_FLAG_SET(name, value) (void)(::testing::GMOCK_FLAG(name) = value)
+
+#endif // GTEST_HAS_ABSL
#endif // GOOGLEMOCK_INCLUDE_GMOCK_INTERNAL_GMOCK_PORT_H_
diff --git a/third_party/googletest/src/googlemock/src/gmock-cardinalities.cc b/third_party/googletest/src/googlemock/src/gmock-cardinalities.cc
index 7463f43..92cde34 100644
--- a/third_party/googletest/src/googlemock/src/gmock-cardinalities.cc
+++ b/third_party/googletest/src/googlemock/src/gmock-cardinalities.cc
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Google Mock - a framework for writing C++ mock classes.
//
// This file implements cardinalities.
@@ -35,9 +34,11 @@
#include "gmock/gmock-cardinalities.h"
#include <limits.h>
+
#include <ostream> // NOLINT
#include <sstream>
#include <string>
+
#include "gmock/internal/gmock-internal-utils.h"
#include "gtest/gtest.h"
@@ -49,8 +50,7 @@
class BetweenCardinalityImpl : public CardinalityInterface {
public:
BetweenCardinalityImpl(int min, int max)
- : min_(min >= 0 ? min : 0),
- max_(max >= min_ ? max : min_) {
+ : min_(min >= 0 ? min : 0), max_(max >= min_ ? max : min_) {
std::stringstream ss;
if (min < 0) {
ss << "The invocation lower bound must be >= 0, "
@@ -62,8 +62,7 @@
internal::Expect(false, __FILE__, __LINE__, ss.str());
} else if (min > max) {
ss << "The invocation upper bound (" << max
- << ") must be >= the invocation lower bound (" << min
- << ").";
+ << ") must be >= the invocation lower bound (" << min << ").";
internal::Expect(false, __FILE__, __LINE__, ss.str());
}
}
@@ -87,7 +86,8 @@
const int min_;
const int max_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(BetweenCardinalityImpl);
+ BetweenCardinalityImpl(const BetweenCardinalityImpl&) = delete;
+ BetweenCardinalityImpl& operator=(const BetweenCardinalityImpl&) = delete;
};
// Formats "n times" in a human-friendly way.
diff --git a/third_party/googletest/src/googlemock/src/gmock-internal-utils.cc b/third_party/googletest/src/googlemock/src/gmock-internal-utils.cc
index e5b5479..0a74841 100644
--- a/third_party/googletest/src/googlemock/src/gmock-internal-utils.cc
+++ b/third_party/googletest/src/googlemock/src/gmock-internal-utils.cc
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Google Mock - a framework for writing C++ mock classes.
//
// This file defines some utilities useful for implementing Google
@@ -37,8 +36,15 @@
#include "gmock/internal/gmock-internal-utils.h"
#include <ctype.h>
+
+#include <array>
+#include <cctype>
+#include <cstdint>
+#include <cstring>
#include <ostream> // NOLINT
#include <string>
+#include <vector>
+
#include "gmock/gmock.h"
#include "gmock/internal/gmock-port.h"
#include "gtest/gtest.h"
@@ -48,21 +54,22 @@
// Joins a vector of strings as if they are fields of a tuple; returns
// the joined string.
-GTEST_API_ std::string JoinAsTuple(const Strings& fields) {
- switch (fields.size()) {
- case 0:
- return "";
- case 1:
- return fields[0];
- default:
- std::string result = "(" + fields[0];
- for (size_t i = 1; i < fields.size(); i++) {
- result += ", ";
- result += fields[i];
- }
- result += ")";
- return result;
+GTEST_API_ std::string JoinAsKeyValueTuple(
+ const std::vector<const char*>& names, const Strings& values) {
+ GTEST_CHECK_(names.size() == values.size());
+ if (values.empty()) {
+ return "";
}
+ const auto build_one = [&](const size_t i) {
+ return std::string(names[i]) + ": " + values[i];
+ };
+ std::string result = "(" + build_one(0);
+ for (size_t i = 1; i < values.size(); i++) {
+ result += ", ";
+ result += build_one(i);
+ }
+ result += ")";
+ return result;
}
// Converts an identifier name to a space-separated list of lower-case
@@ -76,12 +83,11 @@
// We don't care about the current locale as the input is
// guaranteed to be a valid C++ identifier name.
const bool starts_new_word = IsUpper(*p) ||
- (!IsAlpha(prev_char) && IsLower(*p)) ||
- (!IsDigit(prev_char) && IsDigit(*p));
+ (!IsAlpha(prev_char) && IsLower(*p)) ||
+ (!IsDigit(prev_char) && IsDigit(*p));
if (IsAlNum(*p)) {
- if (starts_new_word && result != "")
- result += ' ';
+ if (starts_new_word && result != "") result += ' ';
result += ToLower(*p);
}
}
@@ -95,12 +101,9 @@
public:
void ReportFailure(FailureType type, const char* file, int line,
const std::string& message) override {
- AssertHelper(type == kFatal ?
- TestPartResult::kFatalFailure :
- TestPartResult::kNonFatalFailure,
- file,
- line,
- message.c_str()) = Message();
+ AssertHelper(type == kFatal ? TestPartResult::kFatalFailure
+ : TestPartResult::kNonFatalFailure,
+ file, line, message.c_str()) = Message();
if (type == kFatal) {
posix::Abort();
}
@@ -126,10 +129,10 @@
// Returns true if and only if a log with the given severity is visible
// according to the --gmock_verbose flag.
GTEST_API_ bool LogIsVisible(LogSeverity severity) {
- if (GMOCK_FLAG(verbose) == kInfoVerbosity) {
+ if (GMOCK_FLAG_GET(verbose) == kInfoVerbosity) {
// Always show the log if --gmock_verbose=info.
return true;
- } else if (GMOCK_FLAG(verbose) == kErrorVerbosity) {
+ } else if (GMOCK_FLAG_GET(verbose) == kErrorVerbosity) {
// Always hide it if --gmock_verbose=error.
return false;
} else {
@@ -148,8 +151,7 @@
// conservative.
GTEST_API_ void Log(LogSeverity severity, const std::string& message,
int stack_frames_to_skip) {
- if (!LogIsVisible(severity))
- return;
+ if (!LogIsVisible(severity)) return;
// Ensures that logs from different threads don't interleave.
MutexLock l(&g_log_mutex);
@@ -178,8 +180,8 @@
std::cout << "\n";
}
std::cout << "Stack trace:\n"
- << ::testing::internal::GetCurrentOsStackTraceExceptTop(
- ::testing::UnitTest::GetInstance(), actual_to_skip);
+ << ::testing::internal::GetCurrentOsStackTraceExceptTop(
+ ::testing::UnitTest::GetInstance(), actual_to_skip);
}
std::cout << ::std::flush;
}
@@ -196,5 +198,53 @@
"the variable in various places.");
}
+constexpr char UnBase64Impl(char c, const char* const base64, char carry) {
+ return *base64 == 0 ? static_cast<char>(65)
+ : *base64 == c ? carry
+ : UnBase64Impl(c, base64 + 1, carry + 1);
+}
+
+template <size_t... I>
+constexpr std::array<char, 256> UnBase64Impl(IndexSequence<I...>,
+ const char* const base64) {
+ return {{UnBase64Impl(static_cast<char>(I), base64, 0)...}};
+}
+
+constexpr std::array<char, 256> UnBase64(const char* const base64) {
+ return UnBase64Impl(MakeIndexSequence<256>{}, base64);
+}
+
+static constexpr char kBase64[] =
+ "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
+static constexpr std::array<char, 256> kUnBase64 = UnBase64(kBase64);
+
+bool Base64Unescape(const std::string& encoded, std::string* decoded) {
+ decoded->clear();
+ size_t encoded_len = encoded.size();
+ decoded->reserve(3 * (encoded_len / 4) + (encoded_len % 4));
+ int bit_pos = 0;
+ char dst = 0;
+ for (int src : encoded) {
+ if (std::isspace(src) || src == '=') {
+ continue;
+ }
+ char src_bin = kUnBase64[static_cast<size_t>(src)];
+ if (src_bin >= 64) {
+ decoded->clear();
+ return false;
+ }
+ if (bit_pos == 0) {
+ dst |= static_cast<char>(src_bin << 2);
+ bit_pos = 6;
+ } else {
+ dst |= static_cast<char>(src_bin >> (bit_pos - 2));
+ decoded->push_back(dst);
+ dst = static_cast<char>(src_bin << (10 - bit_pos));
+ bit_pos = (bit_pos + 6) % 8;
+ }
+ }
+ return true;
+}
+
} // namespace internal
} // namespace testing
diff --git a/third_party/googletest/src/googlemock/src/gmock-matchers.cc b/third_party/googletest/src/googlemock/src/gmock-matchers.cc
index dded437..a8d04a6 100644
--- a/third_party/googletest/src/googlemock/src/gmock-matchers.cc
+++ b/third_party/googletest/src/googlemock/src/gmock-matchers.cc
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Google Mock - a framework for writing C++ mock classes.
//
// This file implements Matcher<const string&>, Matcher<string>, and
@@ -36,9 +35,11 @@
#include "gmock/gmock-matchers.h"
#include <string.h>
+
#include <iostream>
#include <sstream>
#include <string>
+#include <vector>
namespace testing {
namespace internal {
@@ -48,11 +49,13 @@
// 'negation' is false; otherwise returns the description of the
// negation of the matcher. 'param_values' contains a list of strings
// that are the print-out of the matcher's parameters.
-GTEST_API_ std::string FormatMatcherDescription(bool negation,
- const char* matcher_name,
- const Strings& param_values) {
+GTEST_API_ std::string FormatMatcherDescription(
+ bool negation, const char* matcher_name,
+ const std::vector<const char*>& param_names, const Strings& param_values) {
std::string result = ConvertIdentifierNameToWords(matcher_name);
- if (param_values.size() >= 1) result += " " + JoinAsTuple(param_values);
+ if (param_values.size() >= 1) {
+ result += " " + JoinAsKeyValueTuple(param_names, param_values);
+ }
return negation ? "not (" + result + ")" : result;
}
diff --git a/third_party/googletest/src/googlemock/src/gmock-spec-builders.cc b/third_party/googletest/src/googlemock/src/gmock-spec-builders.cc
index c7266a3..658ad3f 100644
--- a/third_party/googletest/src/googlemock/src/gmock-spec-builders.cc
+++ b/third_party/googletest/src/googlemock/src/gmock-spec-builders.cc
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Google Mock - a framework for writing C++ mock classes.
//
// This file implements the spec builder syntax (ON_CALL and
@@ -42,6 +41,7 @@
#include <memory>
#include <set>
#include <string>
+#include <unordered_map>
#include <vector>
#include "gmock/gmock.h"
@@ -49,15 +49,15 @@
#include "gtest/internal/gtest-port.h"
#if GTEST_OS_CYGWIN || GTEST_OS_LINUX || GTEST_OS_MAC
-# include <unistd.h> // NOLINT
+#include <unistd.h> // NOLINT
#endif
// Silence C4800 (C4800: 'int *const ': forcing value
// to bool 'true' or 'false') for MSVC 15
#ifdef _MSC_VER
#if _MSC_VER == 1900
-# pragma warning(push)
-# pragma warning(disable:4800)
+#pragma warning(push)
+#pragma warning(disable : 4800)
#endif
#endif
@@ -195,11 +195,12 @@
// Describes the state of the expectation (e.g. is it satisfied?
// is it active?).
- *os << " - " << (IsOverSaturated() ? "over-saturated" :
- IsSaturated() ? "saturated" :
- IsSatisfied() ? "satisfied" : "unsatisfied")
- << " and "
- << (is_retired() ? "retired" : "active");
+ *os << " - "
+ << (IsOverSaturated() ? "over-saturated"
+ : IsSaturated() ? "saturated"
+ : IsSatisfied() ? "satisfied"
+ : "unsatisfied")
+ << " and " << (is_retired() ? "retired" : "active");
}
// Checks the action count (i.e. the number of WillOnce() and
@@ -242,13 +243,12 @@
::std::stringstream ss;
DescribeLocationTo(&ss);
- ss << "Too " << (too_many ? "many" : "few")
- << " actions specified in " << source_text() << "...\n"
+ ss << "Too " << (too_many ? "many" : "few") << " actions specified in "
+ << source_text() << "...\n"
<< "Expected to be ";
cardinality().DescribeTo(&ss);
- ss << ", but has " << (too_many ? "" : "only ")
- << action_count << " WillOnce()"
- << (action_count == 1 ? "" : "s");
+ ss << ", but has " << (too_many ? "" : "only ") << action_count
+ << " WillOnce()" << (action_count == 1 ? "" : "s");
if (repeated_action_specified_) {
ss << " and a WillRepeatedly()";
}
@@ -264,10 +264,10 @@
".Times() cannot appear "
"more than once in an EXPECT_CALL().");
} else {
- ExpectSpecProperty(last_clause_ < kTimes,
- ".Times() cannot appear after "
- ".InSequence(), .WillOnce(), .WillRepeatedly(), "
- "or .RetiresOnSaturation().");
+ ExpectSpecProperty(
+ last_clause_ < kTimes,
+ ".Times() may only appear *before* .InSequence(), .WillOnce(), "
+ ".WillRepeatedly(), or .RetiresOnSaturation(), not after.");
}
last_clause_ = kTimes;
@@ -283,7 +283,7 @@
void ReportUninterestingCall(CallReaction reaction, const std::string& msg) {
// Include a stack trace only if --gmock_verbose=info is specified.
const int stack_frames_to_skip =
- GMOCK_FLAG(verbose) == kInfoVerbosity ? 3 : -1;
+ GMOCK_FLAG_GET(verbose) == kInfoVerbosity ? 3 : -1;
switch (reaction) {
case kAllow:
Log(kInfo, msg, stack_frames_to_skip);
@@ -370,143 +370,12 @@
return name;
}
-// Calculates the result of invoking this mock function with the given
-// arguments, prints it, and returns it. The caller is responsible
-// for deleting the result.
-UntypedActionResultHolderBase* UntypedFunctionMockerBase::UntypedInvokeWith(
- void* const untyped_args) GTEST_LOCK_EXCLUDED_(g_gmock_mutex) {
- // See the definition of untyped_expectations_ for why access to it
- // is unprotected here.
- if (untyped_expectations_.size() == 0) {
- // No expectation is set on this mock method - we have an
- // uninteresting call.
-
- // We must get Google Mock's reaction on uninteresting calls
- // made on this mock object BEFORE performing the action,
- // because the action may DELETE the mock object and make the
- // following expression meaningless.
- const CallReaction reaction =
- Mock::GetReactionOnUninterestingCalls(MockObject());
-
- // True if and only if we need to print this call's arguments and return
- // value. This definition must be kept in sync with
- // the behavior of ReportUninterestingCall().
- const bool need_to_report_uninteresting_call =
- // If the user allows this uninteresting call, we print it
- // only when they want informational messages.
- reaction == kAllow ? LogIsVisible(kInfo) :
- // If the user wants this to be a warning, we print
- // it only when they want to see warnings.
- reaction == kWarn
- ? LogIsVisible(kWarning)
- :
- // Otherwise, the user wants this to be an error, and we
- // should always print detailed information in the error.
- true;
-
- if (!need_to_report_uninteresting_call) {
- // Perform the action without printing the call information.
- return this->UntypedPerformDefaultAction(
- untyped_args, "Function call: " + std::string(Name()));
- }
-
- // Warns about the uninteresting call.
- ::std::stringstream ss;
- this->UntypedDescribeUninterestingCall(untyped_args, &ss);
-
- // Calculates the function result.
- UntypedActionResultHolderBase* const result =
- this->UntypedPerformDefaultAction(untyped_args, ss.str());
-
- // Prints the function result.
- if (result != nullptr) result->PrintAsActionResult(&ss);
-
- ReportUninterestingCall(reaction, ss.str());
- return result;
- }
-
- bool is_excessive = false;
- ::std::stringstream ss;
- ::std::stringstream why;
- ::std::stringstream loc;
- const void* untyped_action = nullptr;
-
- // The UntypedFindMatchingExpectation() function acquires and
- // releases g_gmock_mutex.
-
- const ExpectationBase* const untyped_expectation =
- this->UntypedFindMatchingExpectation(untyped_args, &untyped_action,
- &is_excessive, &ss, &why);
- const bool found = untyped_expectation != nullptr;
-
- // True if and only if we need to print the call's arguments
- // and return value.
- // This definition must be kept in sync with the uses of Expect()
- // and Log() in this function.
- const bool need_to_report_call =
- !found || is_excessive || LogIsVisible(kInfo);
- if (!need_to_report_call) {
- // Perform the action without printing the call information.
- return untyped_action == nullptr
- ? this->UntypedPerformDefaultAction(untyped_args, "")
- : this->UntypedPerformAction(untyped_action, untyped_args);
- }
-
- ss << " Function call: " << Name();
- this->UntypedPrintArgs(untyped_args, &ss);
-
- // In case the action deletes a piece of the expectation, we
- // generate the message beforehand.
- if (found && !is_excessive) {
- untyped_expectation->DescribeLocationTo(&loc);
- }
-
- UntypedActionResultHolderBase* result = nullptr;
-
- auto perform_action = [&] {
- return untyped_action == nullptr
- ? this->UntypedPerformDefaultAction(untyped_args, ss.str())
- : this->UntypedPerformAction(untyped_action, untyped_args);
- };
- auto handle_failures = [&] {
- ss << "\n" << why.str();
-
- if (!found) {
- // No expectation matches this call - reports a failure.
- Expect(false, nullptr, -1, ss.str());
- } else if (is_excessive) {
- // We had an upper-bound violation and the failure message is in ss.
- Expect(false, untyped_expectation->file(), untyped_expectation->line(),
- ss.str());
- } else {
- // We had an expected call and the matching expectation is
- // described in ss.
- Log(kInfo, loc.str() + ss.str(), 2);
- }
- };
-#if GTEST_HAS_EXCEPTIONS
- try {
- result = perform_action();
- } catch (...) {
- handle_failures();
- throw;
- }
-#else
- result = perform_action();
-#endif
-
- if (result != nullptr) result->PrintAsActionResult(&ss);
- handle_failures();
- return result;
-}
-
// Returns an Expectation object that references and co-owns exp,
// which must be an expectation on this mock function.
Expectation UntypedFunctionMockerBase::GetHandleOf(ExpectationBase* exp) {
// See the definition of untyped_expectations_ for why access to it
// is unprotected here.
- for (UntypedExpectations::const_iterator it =
- untyped_expectations_.begin();
+ for (UntypedExpectations::const_iterator it = untyped_expectations_.begin();
it != untyped_expectations_.end(); ++it) {
if (it->get() == exp) {
return Expectation(*it);
@@ -526,8 +395,7 @@
GTEST_EXCLUSIVE_LOCK_REQUIRED_(g_gmock_mutex) {
g_gmock_mutex.AssertHeld();
bool expectations_met = true;
- for (UntypedExpectations::const_iterator it =
- untyped_expectations_.begin();
+ for (UntypedExpectations::const_iterator it = untyped_expectations_.begin();
it != untyped_expectations_.end(); ++it) {
ExpectationBase* const untyped_expectation = it->get();
if (untyped_expectation->IsOverSaturated()) {
@@ -538,15 +406,15 @@
} else if (!untyped_expectation->IsSatisfied()) {
expectations_met = false;
::std::stringstream ss;
- ss << "Actual function call count doesn't match "
- << untyped_expectation->source_text() << "...\n";
+ ss << "Actual function call count doesn't match "
+ << untyped_expectation->source_text() << "...\n";
// No need to show the source file location of the expectation
// in the description, as the Expect() call that follows already
// takes care of it.
untyped_expectation->MaybeDescribeExtraMatcherTo(&ss);
untyped_expectation->DescribeCallCountTo(&ss);
- Expect(false, untyped_expectation->file(),
- untyped_expectation->line(), ss.str());
+ Expect(false, untyped_expectation->file(), untyped_expectation->line(),
+ ss.str());
}
}
@@ -613,8 +481,7 @@
// object alive. Therefore we report any living object as test
// failure, unless the user explicitly asked us to ignore it.
~MockObjectRegistry() {
- if (!GMOCK_FLAG(catch_leaked_mocks))
- return;
+ if (!GMOCK_FLAG_GET(catch_leaked_mocks)) return;
int leaked_count = 0;
for (StateMap::const_iterator it = states_.begin(); it != states_.end();
@@ -634,7 +501,7 @@
<< state.first_used_test << ")";
}
std::cout << " should be deleted but never is. Its address is @"
- << it->first << ".";
+ << it->first << ".";
leaked_count++;
}
if (leaked_count > 0) {
@@ -668,57 +535,63 @@
// Maps a mock object to the reaction Google Mock should have when an
// uninteresting method is called. Protected by g_gmock_mutex.
-std::map<const void*, internal::CallReaction> g_uninteresting_call_reaction;
+std::unordered_map<uintptr_t, internal::CallReaction>&
+UninterestingCallReactionMap() {
+ static auto* map = new std::unordered_map<uintptr_t, internal::CallReaction>;
+ return *map;
+}
// Sets the reaction Google Mock should have when an uninteresting
// method of the given mock object is called.
-void SetReactionOnUninterestingCalls(const void* mock_obj,
+void SetReactionOnUninterestingCalls(uintptr_t mock_obj,
internal::CallReaction reaction)
GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex) {
internal::MutexLock l(&internal::g_gmock_mutex);
- g_uninteresting_call_reaction[mock_obj] = reaction;
+ UninterestingCallReactionMap()[mock_obj] = reaction;
}
} // namespace
// Tells Google Mock to allow uninteresting calls on the given mock
// object.
-void Mock::AllowUninterestingCalls(const void* mock_obj)
+void Mock::AllowUninterestingCalls(uintptr_t mock_obj)
GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex) {
SetReactionOnUninterestingCalls(mock_obj, internal::kAllow);
}
// Tells Google Mock to warn the user about uninteresting calls on the
// given mock object.
-void Mock::WarnUninterestingCalls(const void* mock_obj)
+void Mock::WarnUninterestingCalls(uintptr_t mock_obj)
GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex) {
SetReactionOnUninterestingCalls(mock_obj, internal::kWarn);
}
// Tells Google Mock to fail uninteresting calls on the given mock
// object.
-void Mock::FailUninterestingCalls(const void* mock_obj)
+void Mock::FailUninterestingCalls(uintptr_t mock_obj)
GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex) {
SetReactionOnUninterestingCalls(mock_obj, internal::kFail);
}
// Tells Google Mock the given mock object is being destroyed and its
// entry in the call-reaction table should be removed.
-void Mock::UnregisterCallReaction(const void* mock_obj)
+void Mock::UnregisterCallReaction(uintptr_t mock_obj)
GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex) {
internal::MutexLock l(&internal::g_gmock_mutex);
- g_uninteresting_call_reaction.erase(mock_obj);
+ UninterestingCallReactionMap().erase(static_cast<uintptr_t>(mock_obj));
}
// Returns the reaction Google Mock will have on uninteresting calls
// made on the given mock object.
internal::CallReaction Mock::GetReactionOnUninterestingCalls(
- const void* mock_obj)
- GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex) {
+ const void* mock_obj) GTEST_LOCK_EXCLUDED_(internal::g_gmock_mutex) {
internal::MutexLock l(&internal::g_gmock_mutex);
- return (g_uninteresting_call_reaction.count(mock_obj) == 0) ?
- internal::intToCallReaction(GMOCK_FLAG(default_mock_behavior)) :
- g_uninteresting_call_reaction[mock_obj];
+ return (UninterestingCallReactionMap().count(
+ reinterpret_cast<uintptr_t>(mock_obj)) == 0)
+ ? internal::intToCallReaction(
+ GMOCK_FLAG_GET(default_mock_behavior))
+ : UninterestingCallReactionMap()[reinterpret_cast<uintptr_t>(
+ mock_obj)];
}
// Tells Google Mock to ignore mock_obj when checking for leaked mock
@@ -873,8 +746,8 @@
void Sequence::AddExpectation(const Expectation& expectation) const {
if (*last_expectation_ != expectation) {
if (last_expectation_->expectation_base() != nullptr) {
- expectation.expectation_base()->immediate_prerequisites_
- += *last_expectation_;
+ expectation.expectation_base()->immediate_prerequisites_ +=
+ *last_expectation_;
}
*last_expectation_ = expectation;
}
@@ -903,6 +776,6 @@
#ifdef _MSC_VER
#if _MSC_VER == 1900
-# pragma warning(pop)
+#pragma warning(pop)
#endif
#endif
diff --git a/third_party/googletest/src/googlemock/src/gmock.cc b/third_party/googletest/src/googlemock/src/gmock.cc
index 7bcdb0b..5025656 100644
--- a/third_party/googletest/src/googlemock/src/gmock.cc
+++ b/third_party/googletest/src/googlemock/src/gmock.cc
@@ -27,17 +27,15 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
#include "gmock/gmock.h"
-#include "gmock/internal/gmock-port.h"
-namespace testing {
+#include "gmock/internal/gmock-port.h"
GMOCK_DEFINE_bool_(catch_leaked_mocks, true,
"true if and only if Google Mock should report leaked "
"mock objects as failures.");
-GMOCK_DEFINE_string_(verbose, internal::kWarningVerbosity,
+GMOCK_DEFINE_string_(verbose, testing::internal::kWarningVerbosity,
"Controls how verbose Google Mock's output is."
" Valid values:\n"
" info - prints all messages.\n"
@@ -51,6 +49,7 @@
" 1 - by default, mocks act as NaggyMocks.\n"
" 2 - by default, mocks act as StrictMocks.");
+namespace testing {
namespace internal {
// Parses a string as a command line flag. The string should have the
@@ -59,18 +58,18 @@
//
// Returns the value of the flag, or NULL if the parsing failed.
static const char* ParseGoogleMockFlagValue(const char* str,
- const char* flag,
+ const char* flag_name,
bool def_optional) {
// str and flag must not be NULL.
- if (str == nullptr || flag == nullptr) return nullptr;
+ if (str == nullptr || flag_name == nullptr) return nullptr;
// The flag must start with "--gmock_".
- const std::string flag_str = std::string("--gmock_") + flag;
- const size_t flag_len = flag_str.length();
- if (strncmp(str, flag_str.c_str(), flag_len) != 0) return nullptr;
+ const std::string flag_name_str = std::string("--gmock_") + flag_name;
+ const size_t flag_name_len = flag_name_str.length();
+ if (strncmp(str, flag_name_str.c_str(), flag_name_len) != 0) return nullptr;
// Skips the flag name.
- const char* flag_end = str + flag_len;
+ const char* flag_end = str + flag_name_len;
// When def_optional is true, it's OK to not have a "=value" part.
if (def_optional && (flag_end[0] == '\0')) {
@@ -91,10 +90,10 @@
//
// On success, stores the value of the flag in *value, and returns
// true. On failure, returns false without changing *value.
-static bool ParseGoogleMockBoolFlag(const char* str, const char* flag,
- bool* value) {
+static bool ParseGoogleMockFlag(const char* str, const char* flag_name,
+ bool* value) {
// Gets the value of the flag as a string.
- const char* const value_str = ParseGoogleMockFlagValue(str, flag, true);
+ const char* const value_str = ParseGoogleMockFlagValue(str, flag_name, true);
// Aborts if the parsing failed.
if (value_str == nullptr) return false;
@@ -110,10 +109,10 @@
// On success, stores the value of the flag in *value, and returns
// true. On failure, returns false without changing *value.
template <typename String>
-static bool ParseGoogleMockStringFlag(const char* str, const char* flag,
- String* value) {
+static bool ParseGoogleMockFlag(const char* str, const char* flag_name,
+ String* value) {
// Gets the value of the flag as a string.
- const char* const value_str = ParseGoogleMockFlagValue(str, flag, false);
+ const char* const value_str = ParseGoogleMockFlagValue(str, flag_name, false);
// Aborts if the parsing failed.
if (value_str == nullptr) return false;
@@ -123,17 +122,17 @@
return true;
}
-static bool ParseGoogleMockIntFlag(const char* str, const char* flag,
- int32_t* value) {
+static bool ParseGoogleMockFlag(const char* str, const char* flag_name,
+ int32_t* value) {
// Gets the value of the flag as a string.
- const char* const value_str = ParseGoogleMockFlagValue(str, flag, true);
+ const char* const value_str = ParseGoogleMockFlagValue(str, flag_name, true);
// Aborts if the parsing failed.
if (value_str == nullptr) return false;
// Sets *value to the value of the flag.
- return ParseInt32(Message() << "The value of flag --" << flag,
- value_str, value);
+ return ParseInt32(Message() << "The value of flag --" << flag_name, value_str,
+ value);
}
// The internal implementation of InitGoogleMock().
@@ -152,11 +151,22 @@
const char* const arg = arg_string.c_str();
// Do we see a Google Mock flag?
- if (ParseGoogleMockBoolFlag(arg, "catch_leaked_mocks",
- &GMOCK_FLAG(catch_leaked_mocks)) ||
- ParseGoogleMockStringFlag(arg, "verbose", &GMOCK_FLAG(verbose)) ||
- ParseGoogleMockIntFlag(arg, "default_mock_behavior",
- &GMOCK_FLAG(default_mock_behavior))) {
+ bool found_gmock_flag = false;
+
+#define GMOCK_INTERNAL_PARSE_FLAG(flag_name) \
+ if (!found_gmock_flag) { \
+ auto value = GMOCK_FLAG_GET(flag_name); \
+ if (ParseGoogleMockFlag(arg, #flag_name, &value)) { \
+ GMOCK_FLAG_SET(flag_name, value); \
+ found_gmock_flag = true; \
+ } \
+ }
+
+ GMOCK_INTERNAL_PARSE_FLAG(catch_leaked_mocks)
+ GMOCK_INTERNAL_PARSE_FLAG(verbose)
+ GMOCK_INTERNAL_PARSE_FLAG(default_mock_behavior)
+
+ if (found_gmock_flag) {
// Yes. Shift the remainder of the argv list left by one. Note
// that argv has (*argc + 1) elements, the last one always being
// NULL. The following loop moves the trailing NULL element as
diff --git a/third_party/googletest/src/googlemock/src/gmock_main.cc b/third_party/googletest/src/googlemock/src/gmock_main.cc
index 18c500f..b411c5e 100644
--- a/third_party/googletest/src/googlemock/src/gmock_main.cc
+++ b/third_party/googletest/src/googlemock/src/gmock_main.cc
@@ -27,8 +27,8 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
#include <iostream>
+
#include "gmock/gmock.h"
#include "gtest/gtest.h"
@@ -56,7 +56,7 @@
// https://web.archive.org/web/20170912203238/connect.microsoft.com/VisualStudio/feedback/details/394464/wmain-link-error-in-the-static-library
// // NOLINT
#if GTEST_OS_WINDOWS_MOBILE
-# include <tchar.h> // NOLINT
+#include <tchar.h> // NOLINT
GTEST_API_ int _tmain(int argc, TCHAR** argv) {
#else
diff --git a/third_party/googletest/src/googletest/CMakeLists.txt b/third_party/googletest/src/googletest/CMakeLists.txt
index abdd98b..aa00a5f 100644
--- a/third_party/googletest/src/googletest/CMakeLists.txt
+++ b/third_party/googletest/src/googletest/CMakeLists.txt
@@ -46,14 +46,9 @@
# Project version:
-if (CMAKE_VERSION VERSION_LESS 3.0)
- project(gtest CXX C)
- set(PROJECT_VERSION ${GOOGLETEST_VERSION})
-else()
- cmake_policy(SET CMP0048 NEW)
- project(gtest VERSION ${GOOGLETEST_VERSION} LANGUAGES CXX C)
-endif()
-cmake_minimum_required(VERSION 2.8.12)
+cmake_minimum_required(VERSION 3.5)
+cmake_policy(SET CMP0048 NEW)
+project(gtest VERSION ${GOOGLETEST_VERSION} LANGUAGES CXX C)
if (POLICY CMP0063) # Visibility
cmake_policy(SET CMP0063 NEW)
@@ -136,13 +131,17 @@
# to the targets for when we are part of a parent build (ie being pulled
# in via add_subdirectory() rather than being a standalone build).
if (DEFINED CMAKE_VERSION AND NOT "${CMAKE_VERSION}" VERSION_LESS "2.8.11")
+ string(REPLACE ";" "$<SEMICOLON>" dirs "${gtest_build_include_dirs}")
target_include_directories(gtest SYSTEM INTERFACE
- "$<BUILD_INTERFACE:${gtest_build_include_dirs}>"
+ "$<BUILD_INTERFACE:${dirs}>"
"$<INSTALL_INTERFACE:$<INSTALL_PREFIX>/${CMAKE_INSTALL_INCLUDEDIR}>")
target_include_directories(gtest_main SYSTEM INTERFACE
- "$<BUILD_INTERFACE:${gtest_build_include_dirs}>"
+ "$<BUILD_INTERFACE:${dirs}>"
"$<INSTALL_INTERFACE:$<INSTALL_PREFIX>/${CMAKE_INSTALL_INCLUDEDIR}>")
endif()
+if(CMAKE_SYSTEM_NAME MATCHES "QNX")
+ target_link_libraries(gtest PUBLIC regex)
+endif()
target_link_libraries(gtest_main PUBLIC gtest)
########################################################################
diff --git a/third_party/googletest/src/googletest/README.md b/third_party/googletest/src/googletest/README.md
index 1f8b349..d26b309 100644
--- a/third_party/googletest/src/googletest/README.md
+++ b/third_party/googletest/src/googletest/README.md
@@ -25,7 +25,7 @@
with
```
-git clone https://github.com/google/googletest.git -b release-1.10.0
+git clone https://github.com/google/googletest.git -b release-1.11.0
cd googletest # Main directory of the cloned repository.
mkdir build # Create a directory to hold the build output.
cd build
@@ -94,7 +94,7 @@
FetchContent_Declare(
googletest
# Specify the commit you depend on and update it regularly.
- URL https://github.com/google/googletest/archive/609281088cfefc76f9d0ce82e1ff6c30cc3591e5.zip
+ URL https://github.com/google/googletest/archive/e2239ee6043f73722e7aa812a459f54a28552929.zip
)
# For Windows: Prevent overriding the parent project's compiler/linker settings
set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)
@@ -203,7 +203,9 @@
-DGTEST_DONT_DEFINE_FOO=1
to the compiler flags to tell GoogleTest to change the macro's name from `FOO`
-to `GTEST_FOO`. Currently `FOO` can be `FAIL`, `SUCCEED`, or `TEST`. For
+to `GTEST_FOO`. Currently `FOO` can be `ASSERT_EQ`, `ASSERT_FALSE`, `ASSERT_GE`,
+`ASSERT_GT`, `ASSERT_LE`, `ASSERT_LT`, `ASSERT_NE`, `ASSERT_TRUE`,
+`EXPECT_FALSE`, `EXPECT_TRUE`, `FAIL`, `SUCCEED`, `TEST`, or `TEST_F`. For
example, with `-DGTEST_DONT_DEFINE_TEST=1`, you'll need to write
GTEST_TEST(SomeTest, DoesThis) { ... }
diff --git a/third_party/googletest/src/googletest/cmake/internal_utils.cmake b/third_party/googletest/src/googletest/cmake/internal_utils.cmake
index 8d8d60a..5a34c07 100644
--- a/third_party/googletest/src/googletest/cmake/internal_utils.cmake
+++ b/third_party/googletest/src/googletest/cmake/internal_utils.cmake
@@ -84,13 +84,13 @@
# Ensure MSVC treats source files as UTF-8 encoded.
set(cxx_base_flags "${cxx_base_flags} -utf-8")
elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
- set(cxx_base_flags "-Wall -Wshadow -Werror -Wconversion")
+ set(cxx_base_flags "-Wall -Wshadow -Wconversion")
set(cxx_exception_flags "-fexceptions")
set(cxx_no_exception_flags "-fno-exceptions")
set(cxx_strict_flags "-W -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wredundant-decls")
set(cxx_no_rtti_flags "-fno-rtti")
elseif (CMAKE_COMPILER_IS_GNUCXX)
- set(cxx_base_flags "-Wall -Wshadow -Werror")
+ set(cxx_base_flags "-Wall -Wshadow")
if(NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 7.0.0)
set(cxx_base_flags "${cxx_base_flags} -Wno-error=dangling-else")
endif()
@@ -154,10 +154,6 @@
set_target_properties(${name}
PROPERTIES
COMPILE_FLAGS "${cxx_flags}")
- # Generate debug library name with a postfix.
- set_target_properties(${name}
- PROPERTIES
- DEBUG_POSTFIX "d")
# Set the output directory for build artifacts
set_target_properties(${name}
PROPERTIES
@@ -304,6 +300,8 @@
COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/${name}.py
--build_dir=${CMAKE_CURRENT_BINARY_DIR}/\${CTEST_CONFIGURATION_TYPE} ${ARGN})
endif()
+ # Make the Python import path consistent between Bazel and CMake.
+ set_tests_properties(${name} PROPERTIES ENVIRONMENT PYTHONPATH=${CMAKE_SOURCE_DIR})
endif(PYTHONINTERP_FOUND)
endfunction()
diff --git a/third_party/googletest/src/googletest/include/gtest/gtest-assertion-result.h b/third_party/googletest/src/googletest/include/gtest/gtest-assertion-result.h
new file mode 100644
index 0000000..addbb59
--- /dev/null
+++ b/third_party/googletest/src/googletest/include/gtest/gtest-assertion-result.h
@@ -0,0 +1,237 @@
+// Copyright 2005, Google Inc.
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are
+// met:
+//
+// * Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// * Redistributions in binary form must reproduce the above
+// copyright notice, this list of conditions and the following disclaimer
+// in the documentation and/or other materials provided with the
+// distribution.
+// * Neither the name of Google Inc. nor the names of its
+// contributors may be used to endorse or promote products derived from
+// this software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+// The Google C++ Testing and Mocking Framework (Google Test)
+//
+// This file implements the AssertionResult type.
+
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
+
+#ifndef GOOGLETEST_INCLUDE_GTEST_GTEST_ASSERTION_RESULT_H_
+#define GOOGLETEST_INCLUDE_GTEST_GTEST_ASSERTION_RESULT_H_
+
+#include <memory>
+#include <ostream>
+#include <string>
+#include <type_traits>
+
+#include "gtest/gtest-message.h"
+#include "gtest/internal/gtest-port.h"
+
+GTEST_DISABLE_MSC_WARNINGS_PUSH_(4251 \
+/* class A needs to have dll-interface to be used by clients of class B */)
+
+namespace testing {
+
+// A class for indicating whether an assertion was successful. When
+// the assertion wasn't successful, the AssertionResult object
+// remembers a non-empty message that describes how it failed.
+//
+// To create an instance of this class, use one of the factory functions
+// (AssertionSuccess() and AssertionFailure()).
+//
+// This class is useful for two purposes:
+// 1. Defining predicate functions to be used with Boolean test assertions
+// EXPECT_TRUE/EXPECT_FALSE and their ASSERT_ counterparts
+// 2. Defining predicate-format functions to be
+// used with predicate assertions (ASSERT_PRED_FORMAT*, etc).
+//
+// For example, if you define IsEven predicate:
+//
+// testing::AssertionResult IsEven(int n) {
+// if ((n % 2) == 0)
+// return testing::AssertionSuccess();
+// else
+// return testing::AssertionFailure() << n << " is odd";
+// }
+//
+// Then the failed expectation EXPECT_TRUE(IsEven(Fib(5)))
+// will print the message
+//
+// Value of: IsEven(Fib(5))
+// Actual: false (5 is odd)
+// Expected: true
+//
+// instead of a more opaque
+//
+// Value of: IsEven(Fib(5))
+// Actual: false
+// Expected: true
+//
+// in case IsEven is a simple Boolean predicate.
+//
+// If you expect your predicate to be reused and want to support informative
+// messages in EXPECT_FALSE and ASSERT_FALSE (negative assertions show up
+// about half as often as positive ones in our tests), supply messages for
+// both success and failure cases:
+//
+// testing::AssertionResult IsEven(int n) {
+// if ((n % 2) == 0)
+// return testing::AssertionSuccess() << n << " is even";
+// else
+// return testing::AssertionFailure() << n << " is odd";
+// }
+//
+// Then a statement EXPECT_FALSE(IsEven(Fib(6))) will print
+//
+// Value of: IsEven(Fib(6))
+// Actual: true (8 is even)
+// Expected: false
+//
+// NB: Predicates that support negative Boolean assertions have reduced
+// performance in positive ones so be careful not to use them in tests
+// that have lots (tens of thousands) of positive Boolean assertions.
+//
+// To use this class with EXPECT_PRED_FORMAT assertions such as:
+//
+// // Verifies that Foo() returns an even number.
+// EXPECT_PRED_FORMAT1(IsEven, Foo());
+//
+// you need to define:
+//
+// testing::AssertionResult IsEven(const char* expr, int n) {
+// if ((n % 2) == 0)
+// return testing::AssertionSuccess();
+// else
+// return testing::AssertionFailure()
+// << "Expected: " << expr << " is even\n Actual: it's " << n;
+// }
+//
+// If Foo() returns 5, you will see the following message:
+//
+// Expected: Foo() is even
+// Actual: it's 5
+//
+class GTEST_API_ AssertionResult {
+ public:
+ // Copy constructor.
+ // Used in EXPECT_TRUE/FALSE(assertion_result).
+ AssertionResult(const AssertionResult& other);
+
+// C4800 is a level 3 warning in Visual Studio 2015 and earlier.
+// This warning is not emitted in Visual Studio 2017.
+// This warning is off by default starting in Visual Studio 2019 but can be
+// enabled with command-line options.
+#if defined(_MSC_VER) && (_MSC_VER < 1910 || _MSC_VER >= 1920)
+ GTEST_DISABLE_MSC_WARNINGS_PUSH_(4800 /* forcing value to bool */)
+#endif
+
+ // Used in the EXPECT_TRUE/FALSE(bool_expression).
+ //
+ // T must be contextually convertible to bool.
+ //
+ // The second parameter prevents this overload from being considered if
+ // the argument is implicitly convertible to AssertionResult. In that case
+ // we want AssertionResult's copy constructor to be used.
+ template <typename T>
+ explicit AssertionResult(
+ const T& success,
+ typename std::enable_if<
+ !std::is_convertible<T, AssertionResult>::value>::type*
+ /*enabler*/
+ = nullptr)
+ : success_(success) {}
+
+#if defined(_MSC_VER) && (_MSC_VER < 1910 || _MSC_VER >= 1920)
+ GTEST_DISABLE_MSC_WARNINGS_POP_()
+#endif
+
+ // Assignment operator.
+ AssertionResult& operator=(AssertionResult other) {
+ swap(other);
+ return *this;
+ }
+
+ // Returns true if and only if the assertion succeeded.
+ operator bool() const { return success_; } // NOLINT
+
+ // Returns the assertion's negation. Used with EXPECT/ASSERT_FALSE.
+ AssertionResult operator!() const;
+
+ // Returns the text streamed into this AssertionResult. Test assertions
+ // use it when they fail (i.e., the predicate's outcome doesn't match the
+ // assertion's expectation). When nothing has been streamed into the
+ // object, returns an empty string.
+ const char* message() const {
+ return message_.get() != nullptr ? message_->c_str() : "";
+ }
+ // Deprecated; please use message() instead.
+ const char* failure_message() const { return message(); }
+
+ // Streams a custom failure message into this object.
+ template <typename T>
+ AssertionResult& operator<<(const T& value) {
+ AppendMessage(Message() << value);
+ return *this;
+ }
+
+ // Allows streaming basic output manipulators such as endl or flush into
+ // this object.
+ AssertionResult& operator<<(
+ ::std::ostream& (*basic_manipulator)(::std::ostream& stream)) {
+ AppendMessage(Message() << basic_manipulator);
+ return *this;
+ }
+
+ private:
+ // Appends the contents of message to message_.
+ void AppendMessage(const Message& a_message) {
+ if (message_.get() == nullptr) message_.reset(new ::std::string);
+ message_->append(a_message.GetString().c_str());
+ }
+
+ // Swap the contents of this AssertionResult with other.
+ void swap(AssertionResult& other);
+
+ // Stores result of the assertion predicate.
+ bool success_;
+ // Stores the message describing the condition in case the expectation
+ // construct is not satisfied with the predicate's outcome.
+ // Referenced via a pointer to avoid taking too much stack frame space
+ // with test assertions.
+ std::unique_ptr< ::std::string> message_;
+};
+
+// Makes a successful assertion result.
+GTEST_API_ AssertionResult AssertionSuccess();
+
+// Makes a failed assertion result.
+GTEST_API_ AssertionResult AssertionFailure();
+
+// Makes a failed assertion result with the given failure message.
+// Deprecated; use AssertionFailure() << msg.
+GTEST_API_ AssertionResult AssertionFailure(const Message& msg);
+
+} // namespace testing
+
+GTEST_DISABLE_MSC_WARNINGS_POP_() // 4251
+
+#endif // GOOGLETEST_INCLUDE_GTEST_GTEST_ASSERTION_RESULT_H_
diff --git a/third_party/googletest/src/googletest/include/gtest/gtest-death-test.h b/third_party/googletest/src/googletest/include/gtest/gtest-death-test.h
index 9b4d4d1..84e5a5b 100644
--- a/third_party/googletest/src/googletest/include/gtest/gtest-death-test.h
+++ b/third_party/googletest/src/googletest/include/gtest/gtest-death-test.h
@@ -27,21 +27,21 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
// The Google C++ Testing and Mocking Framework (Google Test)
//
// This header file defines the public API for death tests. It is
// #included by gtest.h so a user doesn't need to include this
// directly.
-// GOOGLETEST_CM0001 DO NOT DELETE
+
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLETEST_INCLUDE_GTEST_GTEST_DEATH_TEST_H_
#define GOOGLETEST_INCLUDE_GTEST_GTEST_DEATH_TEST_H_
#include "gtest/internal/gtest-death-test-internal.h"
-namespace testing {
-
// This flag controls the style of death tests. Valid values are "threadsafe",
// meaning that the death test child process will re-execute the test binary
// from the start, running only a single death test, or "fast",
@@ -49,6 +49,8 @@
// after forking.
GTEST_DECLARE_string_(death_test_style);
+namespace testing {
+
#if GTEST_HAS_DEATH_TEST
namespace internal {
@@ -103,7 +105,6 @@
//
// On the regular expressions used in death tests:
//
-// GOOGLETEST_CM0005 DO NOT DELETE
// On POSIX-compliant systems (*nix), we use the <regex.h> library,
// which uses the POSIX extended regex syntax.
//
@@ -169,24 +170,24 @@
// Asserts that a given `statement` causes the program to exit, with an
// integer exit status that satisfies `predicate`, and emitting error output
// that matches `matcher`.
-# define ASSERT_EXIT(statement, predicate, matcher) \
- GTEST_DEATH_TEST_(statement, predicate, matcher, GTEST_FATAL_FAILURE_)
+#define ASSERT_EXIT(statement, predicate, matcher) \
+ GTEST_DEATH_TEST_(statement, predicate, matcher, GTEST_FATAL_FAILURE_)
// Like `ASSERT_EXIT`, but continues on to successive tests in the
// test suite, if any:
-# define EXPECT_EXIT(statement, predicate, matcher) \
- GTEST_DEATH_TEST_(statement, predicate, matcher, GTEST_NONFATAL_FAILURE_)
+#define EXPECT_EXIT(statement, predicate, matcher) \
+ GTEST_DEATH_TEST_(statement, predicate, matcher, GTEST_NONFATAL_FAILURE_)
// Asserts that a given `statement` causes the program to exit, either by
// explicitly exiting with a nonzero exit code or being killed by a
// signal, and emitting error output that matches `matcher`.
-# define ASSERT_DEATH(statement, matcher) \
- ASSERT_EXIT(statement, ::testing::internal::ExitedUnsuccessfully, matcher)
+#define ASSERT_DEATH(statement, matcher) \
+ ASSERT_EXIT(statement, ::testing::internal::ExitedUnsuccessfully, matcher)
// Like `ASSERT_DEATH`, but continues on to successive tests in the
// test suite, if any:
-# define EXPECT_DEATH(statement, matcher) \
- EXPECT_EXIT(statement, ::testing::internal::ExitedUnsuccessfully, matcher)
+#define EXPECT_DEATH(statement, matcher) \
+ EXPECT_EXIT(statement, ::testing::internal::ExitedUnsuccessfully, matcher)
// Two predicate classes that can be used in {ASSERT,EXPECT}_EXIT*:
@@ -197,22 +198,23 @@
ExitedWithCode(const ExitedWithCode&) = default;
void operator=(const ExitedWithCode& other) = delete;
bool operator()(int exit_status) const;
+
private:
const int exit_code_;
};
-# if !GTEST_OS_WINDOWS && !GTEST_OS_FUCHSIA
+#if !GTEST_OS_WINDOWS && !GTEST_OS_FUCHSIA
// Tests that an exit code describes an exit due to termination by a
// given signal.
-// GOOGLETEST_CM0006 DO NOT DELETE
class GTEST_API_ KilledBySignal {
public:
explicit KilledBySignal(int signum);
bool operator()(int exit_status) const;
+
private:
const int signum_;
};
-# endif // !GTEST_OS_WINDOWS
+#endif // !GTEST_OS_WINDOWS
// EXPECT_DEBUG_DEATH asserts that the given statements die in debug mode.
// The death testing framework causes this to have interesting semantics,
@@ -257,23 +259,21 @@
// EXPECT_EQ(12, DieInDebugOr12(&sideeffect));
// }, "death");
//
-# ifdef NDEBUG
+#ifdef NDEBUG
-# define EXPECT_DEBUG_DEATH(statement, regex) \
+#define EXPECT_DEBUG_DEATH(statement, regex) \
GTEST_EXECUTE_STATEMENT_(statement, regex)
-# define ASSERT_DEBUG_DEATH(statement, regex) \
+#define ASSERT_DEBUG_DEATH(statement, regex) \
GTEST_EXECUTE_STATEMENT_(statement, regex)
-# else
+#else
-# define EXPECT_DEBUG_DEATH(statement, regex) \
- EXPECT_DEATH(statement, regex)
+#define EXPECT_DEBUG_DEATH(statement, regex) EXPECT_DEATH(statement, regex)
-# define ASSERT_DEBUG_DEATH(statement, regex) \
- ASSERT_DEATH(statement, regex)
+#define ASSERT_DEBUG_DEATH(statement, regex) ASSERT_DEATH(statement, regex)
-# endif // NDEBUG for EXPECT_DEBUG_DEATH
+#endif // NDEBUG for EXPECT_DEBUG_DEATH
#endif // GTEST_HAS_DEATH_TEST
// This macro is used for implementing macros such as
@@ -311,18 +311,17 @@
// statement unconditionally returns or throws. The Message constructor at
// the end allows the syntax of streaming additional messages into the
// macro, for compilational compatibility with EXPECT_DEATH/ASSERT_DEATH.
-# define GTEST_UNSUPPORTED_DEATH_TEST(statement, regex, terminator) \
- GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
- if (::testing::internal::AlwaysTrue()) { \
- GTEST_LOG_(WARNING) \
- << "Death tests are not supported on this platform.\n" \
- << "Statement '" #statement "' cannot be verified."; \
- } else if (::testing::internal::AlwaysFalse()) { \
- ::testing::internal::RE::PartialMatch(".*", (regex)); \
- GTEST_SUPPRESS_UNREACHABLE_CODE_WARNING_BELOW_(statement); \
- terminator; \
- } else \
- ::testing::Message()
+#define GTEST_UNSUPPORTED_DEATH_TEST(statement, regex, terminator) \
+ GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
+ if (::testing::internal::AlwaysTrue()) { \
+ GTEST_LOG_(WARNING) << "Death tests are not supported on this platform.\n" \
+ << "Statement '" #statement "' cannot be verified."; \
+ } else if (::testing::internal::AlwaysFalse()) { \
+ ::testing::internal::RE::PartialMatch(".*", (regex)); \
+ GTEST_SUPPRESS_UNREACHABLE_CODE_WARNING_BELOW_(statement); \
+ terminator; \
+ } else \
+ ::testing::Message()
// EXPECT_DEATH_IF_SUPPORTED(statement, regex) and
// ASSERT_DEATH_IF_SUPPORTED(statement, regex) expand to real death tests if
@@ -330,15 +329,15 @@
// useful when you are combining death test assertions with normal test
// assertions in one test.
#if GTEST_HAS_DEATH_TEST
-# define EXPECT_DEATH_IF_SUPPORTED(statement, regex) \
- EXPECT_DEATH(statement, regex)
-# define ASSERT_DEATH_IF_SUPPORTED(statement, regex) \
- ASSERT_DEATH(statement, regex)
+#define EXPECT_DEATH_IF_SUPPORTED(statement, regex) \
+ EXPECT_DEATH(statement, regex)
+#define ASSERT_DEATH_IF_SUPPORTED(statement, regex) \
+ ASSERT_DEATH(statement, regex)
#else
-# define EXPECT_DEATH_IF_SUPPORTED(statement, regex) \
- GTEST_UNSUPPORTED_DEATH_TEST(statement, regex, )
-# define ASSERT_DEATH_IF_SUPPORTED(statement, regex) \
- GTEST_UNSUPPORTED_DEATH_TEST(statement, regex, return)
+#define EXPECT_DEATH_IF_SUPPORTED(statement, regex) \
+ GTEST_UNSUPPORTED_DEATH_TEST(statement, regex, )
+#define ASSERT_DEATH_IF_SUPPORTED(statement, regex) \
+ GTEST_UNSUPPORTED_DEATH_TEST(statement, regex, return)
#endif
} // namespace testing
diff --git a/third_party/googletest/src/googletest/include/gtest/gtest-matchers.h b/third_party/googletest/src/googletest/include/gtest/gtest-matchers.h
index 9fa34a0..bffa00c 100644
--- a/third_party/googletest/src/googletest/include/gtest/gtest-matchers.h
+++ b/third_party/googletest/src/googletest/include/gtest/gtest-matchers.h
@@ -32,6 +32,10 @@
// This file implements just enough of the matcher interface to allow
// EXPECT_DEATH and friends to accept a matcher argument.
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
+
#ifndef GOOGLETEST_INCLUDE_GTEST_GTEST_MATCHERS_H_
#define GOOGLETEST_INCLUDE_GTEST_GTEST_MATCHERS_H_
@@ -98,11 +102,11 @@
private:
::std::ostream* const stream_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(MatchResultListener);
+ MatchResultListener(const MatchResultListener&) = delete;
+ MatchResultListener& operator=(const MatchResultListener&) = delete;
};
-inline MatchResultListener::~MatchResultListener() {
-}
+inline MatchResultListener::~MatchResultListener() {}
// An instance of a subclass of this knows how to describe itself as a
// matcher.
@@ -176,27 +180,39 @@
struct AnyEq {
template <typename A, typename B>
- bool operator()(const A& a, const B& b) const { return a == b; }
+ bool operator()(const A& a, const B& b) const {
+ return a == b;
+ }
};
struct AnyNe {
template <typename A, typename B>
- bool operator()(const A& a, const B& b) const { return a != b; }
+ bool operator()(const A& a, const B& b) const {
+ return a != b;
+ }
};
struct AnyLt {
template <typename A, typename B>
- bool operator()(const A& a, const B& b) const { return a < b; }
+ bool operator()(const A& a, const B& b) const {
+ return a < b;
+ }
};
struct AnyGt {
template <typename A, typename B>
- bool operator()(const A& a, const B& b) const { return a > b; }
+ bool operator()(const A& a, const B& b) const {
+ return a > b;
+ }
};
struct AnyLe {
template <typename A, typename B>
- bool operator()(const A& a, const B& b) const { return a <= b; }
+ bool operator()(const A& a, const B& b) const {
+ return a <= b;
+ }
};
struct AnyGe {
template <typename A, typename B>
- bool operator()(const A& a, const B& b) const { return a >= b; }
+ bool operator()(const A& a, const B& b) const {
+ return a >= b;
+ }
};
// A match result listener that ignores the explanation.
@@ -205,7 +221,8 @@
DummyMatchResultListener() : MatchResultListener(nullptr) {}
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(DummyMatchResultListener);
+ DummyMatchResultListener(const DummyMatchResultListener&) = delete;
+ DummyMatchResultListener& operator=(const DummyMatchResultListener&) = delete;
};
// A match result listener that forwards the explanation to a given
@@ -217,7 +234,9 @@
: MatchResultListener(os) {}
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(StreamMatchResultListener);
+ StreamMatchResultListener(const StreamMatchResultListener&) = delete;
+ StreamMatchResultListener& operator=(const StreamMatchResultListener&) =
+ delete;
};
struct SharedPayloadBase {
@@ -284,17 +303,18 @@
}
protected:
- MatcherBase() : vtable_(nullptr) {}
+ MatcherBase() : vtable_(nullptr), buffer_() {}
// Constructs a matcher from its implementation.
template <typename U>
- explicit MatcherBase(const MatcherInterface<U>* impl) {
+ explicit MatcherBase(const MatcherInterface<U>* impl)
+ : vtable_(nullptr), buffer_() {
Init(impl);
}
template <typename M, typename = typename std::remove_reference<
M>::type::is_gtest_matcher>
- MatcherBase(M&& m) { // NOLINT
+ MatcherBase(M&& m) : vtable_(nullptr), buffer_() { // NOLINT
Init(std::forward<M>(m));
}
@@ -420,8 +440,8 @@
static const M& Get(const MatcherBase& m) {
// When inlined along with Init, need to be explicit to avoid violating
// strict aliasing rules.
- const M *ptr = static_cast<const M*>(
- static_cast<const void*>(&m.buffer_));
+ const M* ptr =
+ static_cast<const M*>(static_cast<const void*>(&m.buffer_));
return *ptr;
}
static void Init(MatcherBase& m, M impl) {
@@ -741,7 +761,7 @@
class EqMatcher : public ComparisonBase<EqMatcher<Rhs>, Rhs, AnyEq> {
public:
explicit EqMatcher(const Rhs& rhs)
- : ComparisonBase<EqMatcher<Rhs>, Rhs, AnyEq>(rhs) { }
+ : ComparisonBase<EqMatcher<Rhs>, Rhs, AnyEq>(rhs) {}
static const char* Desc() { return "is equal to"; }
static const char* NegatedDesc() { return "isn't equal to"; }
};
@@ -749,7 +769,7 @@
class NeMatcher : public ComparisonBase<NeMatcher<Rhs>, Rhs, AnyNe> {
public:
explicit NeMatcher(const Rhs& rhs)
- : ComparisonBase<NeMatcher<Rhs>, Rhs, AnyNe>(rhs) { }
+ : ComparisonBase<NeMatcher<Rhs>, Rhs, AnyNe>(rhs) {}
static const char* Desc() { return "isn't equal to"; }
static const char* NegatedDesc() { return "is equal to"; }
};
@@ -757,7 +777,7 @@
class LtMatcher : public ComparisonBase<LtMatcher<Rhs>, Rhs, AnyLt> {
public:
explicit LtMatcher(const Rhs& rhs)
- : ComparisonBase<LtMatcher<Rhs>, Rhs, AnyLt>(rhs) { }
+ : ComparisonBase<LtMatcher<Rhs>, Rhs, AnyLt>(rhs) {}
static const char* Desc() { return "is <"; }
static const char* NegatedDesc() { return "isn't <"; }
};
@@ -765,7 +785,7 @@
class GtMatcher : public ComparisonBase<GtMatcher<Rhs>, Rhs, AnyGt> {
public:
explicit GtMatcher(const Rhs& rhs)
- : ComparisonBase<GtMatcher<Rhs>, Rhs, AnyGt>(rhs) { }
+ : ComparisonBase<GtMatcher<Rhs>, Rhs, AnyGt>(rhs) {}
static const char* Desc() { return "is >"; }
static const char* NegatedDesc() { return "isn't >"; }
};
@@ -773,7 +793,7 @@
class LeMatcher : public ComparisonBase<LeMatcher<Rhs>, Rhs, AnyLe> {
public:
explicit LeMatcher(const Rhs& rhs)
- : ComparisonBase<LeMatcher<Rhs>, Rhs, AnyLe>(rhs) { }
+ : ComparisonBase<LeMatcher<Rhs>, Rhs, AnyLe>(rhs) {}
static const char* Desc() { return "is <="; }
static const char* NegatedDesc() { return "isn't <="; }
};
@@ -781,7 +801,7 @@
class GeMatcher : public ComparisonBase<GeMatcher<Rhs>, Rhs, AnyGe> {
public:
explicit GeMatcher(const Rhs& rhs)
- : ComparisonBase<GeMatcher<Rhs>, Rhs, AnyGe>(rhs) { }
+ : ComparisonBase<GeMatcher<Rhs>, Rhs, AnyGe>(rhs) {}
static const char* Desc() { return "is >="; }
static const char* NegatedDesc() { return "isn't >="; }
};
@@ -872,12 +892,16 @@
// Note: if the parameter of Eq() were declared as const T&, Eq("foo")
// wouldn't compile.
template <typename T>
-inline internal::EqMatcher<T> Eq(T x) { return internal::EqMatcher<T>(x); }
+inline internal::EqMatcher<T> Eq(T x) {
+ return internal::EqMatcher<T>(x);
+}
// Constructs a Matcher<T> from a 'value' of type T. The constructed
// matcher matches any value that's equal to 'value'.
template <typename T>
-Matcher<T>::Matcher(T value) { *this = Eq(value); }
+Matcher<T>::Matcher(T value) {
+ *this = Eq(value);
+}
// Creates a monomorphic matcher that matches anything with type Lhs
// and equal to rhs. A user may need to use this instead of Eq(...)
@@ -892,7 +916,9 @@
// can always write Matcher<T>(Lt(5)) to be explicit about the type,
// for example.
template <typename Lhs, typename Rhs>
-inline Matcher<Lhs> TypedEq(const Rhs& rhs) { return Eq(rhs); }
+inline Matcher<Lhs> TypedEq(const Rhs& rhs) {
+ return Eq(rhs);
+}
// Creates a polymorphic matcher that matches anything >= x.
template <typename Rhs>
diff --git a/third_party/googletest/src/googletest/include/gtest/gtest-message.h b/third_party/googletest/src/googletest/include/gtest/gtest-message.h
index becfd49..6c8bf90 100644
--- a/third_party/googletest/src/googletest/include/gtest/gtest-message.h
+++ b/third_party/googletest/src/googletest/include/gtest/gtest-message.h
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
// The Google C++ Testing and Mocking Framework (Google Test)
//
// This header file defines the Message class.
@@ -42,7 +41,9 @@
// to CHANGE WITHOUT NOTICE. Therefore DO NOT DEPEND ON IT in a user
// program!
-// GOOGLETEST_CM0001 DO NOT DELETE
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLETEST_INCLUDE_GTEST_GTEST_MESSAGE_H_
#define GOOGLETEST_INCLUDE_GTEST_GTEST_MESSAGE_H_
@@ -110,8 +111,8 @@
// Streams a non-pointer value to this object.
template <typename T>
- inline Message& operator <<(const T& val) {
- // Some libraries overload << for STL containers. These
+ inline Message& operator<<(const T& val) {
+ // Some libraries overload << for STL containers. These
// overloads are defined in the global namespace instead of ::std.
//
// C++'s symbol lookup rule (i.e. Koenig lookup) says that these
@@ -125,7 +126,7 @@
// from the global namespace. With this using declaration,
// overloads of << defined in the global namespace and those
// visible via Koenig lookup are both exposed in this function.
- using ::operator <<;
+ using ::operator<<;
*ss_ << val;
return *this;
}
@@ -144,7 +145,7 @@
// ensure consistent result across compilers, we always treat NULL
// as "(null)".
template <typename T>
- inline Message& operator <<(T* const& pointer) { // NOLINT
+ inline Message& operator<<(T* const& pointer) { // NOLINT
if (pointer == nullptr) {
*ss_ << "(null)";
} else {
@@ -159,25 +160,23 @@
// templatized version above. Without this definition, streaming
// endl or other basic IO manipulators to Message will confuse the
// compiler.
- Message& operator <<(BasicNarrowIoManip val) {
+ Message& operator<<(BasicNarrowIoManip val) {
*ss_ << val;
return *this;
}
// Instead of 1/0, we want to see true/false for bool values.
- Message& operator <<(bool b) {
- return *this << (b ? "true" : "false");
- }
+ Message& operator<<(bool b) { return *this << (b ? "true" : "false"); }
// These two overloads allow streaming a wide C string to a Message
// using the UTF-8 encoding.
- Message& operator <<(const wchar_t* wide_c_str);
- Message& operator <<(wchar_t* wide_c_str);
+ Message& operator<<(const wchar_t* wide_c_str);
+ Message& operator<<(wchar_t* wide_c_str);
#if GTEST_HAS_STD_WSTRING
// Converts the given wide string to a narrow string using the UTF-8
// encoding, and streams the result to this Message object.
- Message& operator <<(const ::std::wstring& wstr);
+ Message& operator<<(const ::std::wstring& wstr);
#endif // GTEST_HAS_STD_WSTRING
// Gets the text streamed to this object so far as an std::string.
@@ -196,7 +195,7 @@
};
// Streams a Message to an ostream.
-inline std::ostream& operator <<(std::ostream& os, const Message& sb) {
+inline std::ostream& operator<<(std::ostream& os, const Message& sb) {
return os << sb.GetString();
}
diff --git a/third_party/googletest/src/googletest/include/gtest/gtest-param-test.h b/third_party/googletest/src/googletest/include/gtest/gtest-param-test.h
index 804e702..b55119a 100644
--- a/third_party/googletest/src/googletest/include/gtest/gtest-param-test.h
+++ b/third_party/googletest/src/googletest/include/gtest/gtest-param-test.h
@@ -26,11 +26,14 @@
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
+
// Macros and functions for implementing parameterized tests
// in Google C++ Testing and Mocking Framework (Google Test)
-//
-// GOOGLETEST_CM0001 DO NOT DELETE
+
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
+
#ifndef GOOGLETEST_INCLUDE_GTEST_GTEST_PARAM_TEST_H_
#define GOOGLETEST_INCLUDE_GTEST_GTEST_PARAM_TEST_H_
@@ -353,9 +356,7 @@
// }
// INSTANTIATE_TEST_SUITE_P(BoolSequence, FlagDependentTest, Bool());
//
-inline internal::ParamGenerator<bool> Bool() {
- return Values(false, true);
-}
+inline internal::ParamGenerator<bool> Bool() { return Values(false, true); }
// Combine() allows the user to combine two or more sequences to produce
// values of a Cartesian product of those sequences' elements.
@@ -428,8 +429,11 @@
return 0; \
} \
static int gtest_registering_dummy_ GTEST_ATTRIBUTE_UNUSED_; \
- GTEST_DISALLOW_COPY_AND_ASSIGN_(GTEST_TEST_CLASS_NAME_(test_suite_name, \
- test_name)); \
+ GTEST_TEST_CLASS_NAME_(test_suite_name, test_name) \
+ (const GTEST_TEST_CLASS_NAME_(test_suite_name, test_name) &) = delete; \
+ GTEST_TEST_CLASS_NAME_(test_suite_name, test_name) & operator=( \
+ const GTEST_TEST_CLASS_NAME_(test_suite_name, \
+ test_name) &) = delete; /* NOLINT */ \
}; \
int GTEST_TEST_CLASS_NAME_(test_suite_name, \
test_name)::gtest_registering_dummy_ = \
@@ -453,43 +457,42 @@
#define GTEST_GET_FIRST_(first, ...) first
#define GTEST_GET_SECOND_(first, second, ...) second
-#define INSTANTIATE_TEST_SUITE_P(prefix, test_suite_name, ...) \
- static ::testing::internal::ParamGenerator<test_suite_name::ParamType> \
- gtest_##prefix##test_suite_name##_EvalGenerator_() { \
- return GTEST_EXPAND_(GTEST_GET_FIRST_(__VA_ARGS__, DUMMY_PARAM_)); \
- } \
- static ::std::string gtest_##prefix##test_suite_name##_EvalGenerateName_( \
- const ::testing::TestParamInfo<test_suite_name::ParamType>& info) { \
- if (::testing::internal::AlwaysFalse()) { \
- ::testing::internal::TestNotEmpty(GTEST_EXPAND_(GTEST_GET_SECOND_( \
- __VA_ARGS__, \
- ::testing::internal::DefaultParamName<test_suite_name::ParamType>, \
- DUMMY_PARAM_))); \
- auto t = std::make_tuple(__VA_ARGS__); \
- static_assert(std::tuple_size<decltype(t)>::value <= 2, \
- "Too Many Args!"); \
- } \
- return ((GTEST_EXPAND_(GTEST_GET_SECOND_( \
- __VA_ARGS__, \
- ::testing::internal::DefaultParamName<test_suite_name::ParamType>, \
- DUMMY_PARAM_))))(info); \
- } \
- static int gtest_##prefix##test_suite_name##_dummy_ \
- GTEST_ATTRIBUTE_UNUSED_ = \
- ::testing::UnitTest::GetInstance() \
- ->parameterized_test_registry() \
- .GetTestSuitePatternHolder<test_suite_name>( \
- GTEST_STRINGIFY_(test_suite_name), \
- ::testing::internal::CodeLocation(__FILE__, __LINE__)) \
- ->AddTestSuiteInstantiation( \
- GTEST_STRINGIFY_(prefix), \
- >est_##prefix##test_suite_name##_EvalGenerator_, \
- >est_##prefix##test_suite_name##_EvalGenerateName_, \
+#define INSTANTIATE_TEST_SUITE_P(prefix, test_suite_name, ...) \
+ static ::testing::internal::ParamGenerator<test_suite_name::ParamType> \
+ gtest_##prefix##test_suite_name##_EvalGenerator_() { \
+ return GTEST_EXPAND_(GTEST_GET_FIRST_(__VA_ARGS__, DUMMY_PARAM_)); \
+ } \
+ static ::std::string gtest_##prefix##test_suite_name##_EvalGenerateName_( \
+ const ::testing::TestParamInfo<test_suite_name::ParamType>& info) { \
+ if (::testing::internal::AlwaysFalse()) { \
+ ::testing::internal::TestNotEmpty(GTEST_EXPAND_(GTEST_GET_SECOND_( \
+ __VA_ARGS__, \
+ ::testing::internal::DefaultParamName<test_suite_name::ParamType>, \
+ DUMMY_PARAM_))); \
+ auto t = std::make_tuple(__VA_ARGS__); \
+ static_assert(std::tuple_size<decltype(t)>::value <= 2, \
+ "Too Many Args!"); \
+ } \
+ return ((GTEST_EXPAND_(GTEST_GET_SECOND_( \
+ __VA_ARGS__, \
+ ::testing::internal::DefaultParamName<test_suite_name::ParamType>, \
+ DUMMY_PARAM_))))(info); \
+ } \
+ static int gtest_##prefix##test_suite_name##_dummy_ \
+ GTEST_ATTRIBUTE_UNUSED_ = \
+ ::testing::UnitTest::GetInstance() \
+ ->parameterized_test_registry() \
+ .GetTestSuitePatternHolder<test_suite_name>( \
+ GTEST_STRINGIFY_(test_suite_name), \
+ ::testing::internal::CodeLocation(__FILE__, __LINE__)) \
+ ->AddTestSuiteInstantiation( \
+ GTEST_STRINGIFY_(prefix), \
+ >est_##prefix##test_suite_name##_EvalGenerator_, \
+ >est_##prefix##test_suite_name##_EvalGenerateName_, \
__FILE__, __LINE__)
-
// Allow Marking a Parameterized test class as not needing to be instantiated.
-#define GTEST_ALLOW_UNINSTANTIATED_PARAMETERIZED_TEST(T) \
+#define GTEST_ALLOW_UNINSTANTIATED_PARAMETERIZED_TEST(T) \
namespace gtest_do_not_use_outside_namespace_scope {} \
static const ::testing::internal::MarkAsIgnored gtest_allow_ignore_##T( \
GTEST_STRINGIFY_(T))
diff --git a/third_party/googletest/src/googletest/include/gtest/gtest-printers.h b/third_party/googletest/src/googletest/include/gtest/gtest-printers.h
index 076c9de..a91e8b8 100644
--- a/third_party/googletest/src/googletest/include/gtest/gtest-printers.h
+++ b/third_party/googletest/src/googletest/include/gtest/gtest-printers.h
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Google Test - The Google C++ Testing and Mocking Framework
//
// This file implements a universal value printer that can print a
@@ -95,7 +94,9 @@
// being defined as many user-defined container types don't have
// value_type.
-// GOOGLETEST_CM0001 DO NOT DELETE
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLETEST_INCLUDE_GTEST_GTEST_PRINTERS_H_
#define GOOGLETEST_INCLUDE_GTEST_GTEST_PRINTERS_H_
@@ -257,12 +258,10 @@
#endif
};
-
// Prints the given number of bytes in the given object to the given
// ostream.
GTEST_API_ void PrintBytesInObjectTo(const unsigned char* obj_bytes,
- size_t count,
- ::std::ostream* os);
+ size_t count, ::std::ostream* os);
struct RawBytesPrinter {
// SFINAE on `sizeof` to make sure we have a complete type.
template <typename T, size_t = sizeof(T)>
@@ -360,7 +359,7 @@
GTEST_IMPL_FORMAT_C_STRING_AS_POINTER_(const char);
GTEST_IMPL_FORMAT_C_STRING_AS_POINTER_(wchar_t);
GTEST_IMPL_FORMAT_C_STRING_AS_POINTER_(const wchar_t);
-#ifdef __cpp_char8_t
+#ifdef __cpp_lib_char8_t
GTEST_IMPL_FORMAT_C_STRING_AS_POINTER_(char8_t);
GTEST_IMPL_FORMAT_C_STRING_AS_POINTER_(const char8_t);
#endif
@@ -375,12 +374,12 @@
// to point to a NUL-terminated string, and thus can print it as a string.
#define GTEST_IMPL_FORMAT_C_STRING_AS_STRING_(CharType, OtherStringType) \
- template <> \
- class FormatForComparison<CharType*, OtherStringType> { \
- public: \
- static ::std::string Format(CharType* value) { \
- return ::testing::PrintToString(value); \
- } \
+ template <> \
+ class FormatForComparison<CharType*, OtherStringType> { \
+ public: \
+ static ::std::string Format(CharType* value) { \
+ return ::testing::PrintToString(value); \
+ } \
}
GTEST_IMPL_FORMAT_C_STRING_AS_STRING_(char, ::std::string);
@@ -410,8 +409,8 @@
//
// INTERNAL IMPLEMENTATION - DO NOT USE IN A USER PROGRAM.
template <typename T1, typename T2>
-std::string FormatForComparisonFailureMessage(
- const T1& value, const T2& /* other_operand */) {
+std::string FormatForComparisonFailureMessage(const T1& value,
+ const T2& /* other_operand */) {
return FormatForComparison<T1, T2>::Format(value);
}
@@ -479,6 +478,12 @@
}
#endif
+// gcc/clang __{u,}int128_t
+#if defined(__SIZEOF_INT128__)
+GTEST_API_ void PrintTo(__uint128_t v, ::std::ostream* os);
+GTEST_API_ void PrintTo(__int128_t v, ::std::ostream* os);
+#endif // __SIZEOF_INT128__
+
// Overloads for C strings.
GTEST_API_ void PrintTo(const char* s, ::std::ostream* os);
inline void PrintTo(char* s, ::std::ostream* os) {
@@ -545,7 +550,7 @@
}
// Overloads for ::std::string.
-GTEST_API_ void PrintStringTo(const ::std::string&s, ::std::ostream* os);
+GTEST_API_ void PrintStringTo(const ::std::string& s, ::std::ostream* os);
inline void PrintTo(const ::std::string& s, ::std::ostream* os) {
PrintStringTo(s, os);
}
@@ -572,7 +577,7 @@
// Overloads for ::std::wstring.
#if GTEST_HAS_STD_WSTRING
-GTEST_API_ void PrintWideStringTo(const ::std::wstring&s, ::std::ostream* os);
+GTEST_API_ void PrintWideStringTo(const ::std::wstring& s, ::std::ostream* os);
inline void PrintTo(const ::std::wstring& s, ::std::ostream* os) {
PrintWideStringTo(s, os);
}
@@ -587,6 +592,12 @@
inline void PrintTo(std::nullptr_t, ::std::ostream* os) { *os << "(nullptr)"; }
+#if GTEST_HAS_RTTI
+inline void PrintTo(const std::type_info& info, std::ostream* os) {
+ *os << internal::GetTypeName(info);
+}
+#endif // GTEST_HAS_RTTI
+
template <typename T>
void PrintTo(std::reference_wrapper<T> ref, ::std::ostream* os) {
UniversalPrinter<T&>::Print(ref.get(), os);
@@ -744,6 +755,14 @@
}
};
+template <>
+class UniversalPrinter<decltype(Nullopt())> {
+ public:
+ static void Print(decltype(Nullopt()), ::std::ostream* os) {
+ *os << "(nullopt)";
+ }
+};
+
#endif // GTEST_INTERNAL_HAS_OPTIONAL
#if GTEST_INTERNAL_HAS_VARIANT
@@ -802,8 +821,8 @@
}
}
// This overload prints a (const) char array compactly.
-GTEST_API_ void UniversalPrintArray(
- const char* begin, size_t len, ::std::ostream* os);
+GTEST_API_ void UniversalPrintArray(const char* begin, size_t len,
+ ::std::ostream* os);
#ifdef __cpp_char8_t
// This overload prints a (const) char8_t array compactly.
@@ -820,8 +839,8 @@
::std::ostream* os);
// This overload prints a (const) wchar_t array compactly.
-GTEST_API_ void UniversalPrintArray(
- const wchar_t* begin, size_t len, ::std::ostream* os);
+GTEST_API_ void UniversalPrintArray(const wchar_t* begin, size_t len,
+ ::std::ostream* os);
// Implements printing an array type T[N].
template <typename T, size_t N>
@@ -980,10 +999,10 @@
UniversalPrinter<T1>::Print(value, os);
}
-typedef ::std::vector< ::std::string> Strings;
+typedef ::std::vector<::std::string> Strings;
- // Tersely prints the first N fields of a tuple to a string vector,
- // one element for each field.
+// Tersely prints the first N fields of a tuple to a string vector,
+// one element for each field.
template <typename Tuple>
void TersePrintPrefixToStrings(const Tuple&, std::integral_constant<size_t, 0>,
Strings*) {}
diff --git a/third_party/googletest/src/googletest/include/gtest/gtest-spi.h b/third_party/googletest/src/googletest/include/gtest/gtest-spi.h
index eacef44..bec8c48 100644
--- a/third_party/googletest/src/googletest/include/gtest/gtest-spi.h
+++ b/third_party/googletest/src/googletest/include/gtest/gtest-spi.h
@@ -27,12 +27,9 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
// Utilities for testing Google Test itself and code that uses Google Test
// (e.g. frameworks built on top of Google Test).
-// GOOGLETEST_CM0004 DO NOT DELETE
-
#ifndef GOOGLETEST_INCLUDE_GTEST_GTEST_SPI_H_
#define GOOGLETEST_INCLUDE_GTEST_GTEST_SPI_H_
@@ -88,7 +85,10 @@
TestPartResultReporterInterface* old_reporter_;
TestPartResultArray* const result_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ScopedFakeTestPartResultReporter);
+ ScopedFakeTestPartResultReporter(const ScopedFakeTestPartResultReporter&) =
+ delete;
+ ScopedFakeTestPartResultReporter& operator=(
+ const ScopedFakeTestPartResultReporter&) = delete;
};
namespace internal {
@@ -104,12 +104,14 @@
SingleFailureChecker(const TestPartResultArray* results,
TestPartResult::Type type, const std::string& substr);
~SingleFailureChecker();
+
private:
const TestPartResultArray* const results_;
const TestPartResult::Type type_;
const std::string substr_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(SingleFailureChecker);
+ SingleFailureChecker(const SingleFailureChecker&) = delete;
+ SingleFailureChecker& operator=(const SingleFailureChecker&) = delete;
};
} // namespace internal
@@ -119,7 +121,8 @@
GTEST_DISABLE_MSC_WARNINGS_POP_() // 4251
// A set of macros for testing Google Test assertions or code that's expected
-// to generate Google Test fatal failures. It verifies that the given
+// to generate Google Test fatal failures (e.g. a failure from an ASSERT_EQ, but
+// not a non-fatal failure, as from EXPECT_EQ). It verifies that the given
// statement will cause exactly one fatal Google Test failure with 'substr'
// being part of the failure message.
//
@@ -141,44 +144,46 @@
// helper macro, due to some peculiarity in how the preprocessor
// works. The AcceptsMacroThatExpandsToUnprotectedComma test in
// gtest_unittest.cc will fail to compile if we do that.
-#define EXPECT_FATAL_FAILURE(statement, substr) \
- do { \
- class GTestExpectFatalFailureHelper {\
- public:\
- static void Execute() { statement; }\
- };\
- ::testing::TestPartResultArray gtest_failures;\
- ::testing::internal::SingleFailureChecker gtest_checker(\
- >est_failures, ::testing::TestPartResult::kFatalFailure, (substr));\
- {\
- ::testing::ScopedFakeTestPartResultReporter gtest_reporter(\
- ::testing::ScopedFakeTestPartResultReporter:: \
- INTERCEPT_ONLY_CURRENT_THREAD, >est_failures);\
- GTestExpectFatalFailureHelper::Execute();\
- }\
+#define EXPECT_FATAL_FAILURE(statement, substr) \
+ do { \
+ class GTestExpectFatalFailureHelper { \
+ public: \
+ static void Execute() { statement; } \
+ }; \
+ ::testing::TestPartResultArray gtest_failures; \
+ ::testing::internal::SingleFailureChecker gtest_checker( \
+ >est_failures, ::testing::TestPartResult::kFatalFailure, (substr)); \
+ { \
+ ::testing::ScopedFakeTestPartResultReporter gtest_reporter( \
+ ::testing::ScopedFakeTestPartResultReporter:: \
+ INTERCEPT_ONLY_CURRENT_THREAD, \
+ >est_failures); \
+ GTestExpectFatalFailureHelper::Execute(); \
+ } \
} while (::testing::internal::AlwaysFalse())
-#define EXPECT_FATAL_FAILURE_ON_ALL_THREADS(statement, substr) \
- do { \
- class GTestExpectFatalFailureHelper {\
- public:\
- static void Execute() { statement; }\
- };\
- ::testing::TestPartResultArray gtest_failures;\
- ::testing::internal::SingleFailureChecker gtest_checker(\
- >est_failures, ::testing::TestPartResult::kFatalFailure, (substr));\
- {\
- ::testing::ScopedFakeTestPartResultReporter gtest_reporter(\
- ::testing::ScopedFakeTestPartResultReporter:: \
- INTERCEPT_ALL_THREADS, >est_failures);\
- GTestExpectFatalFailureHelper::Execute();\
- }\
+#define EXPECT_FATAL_FAILURE_ON_ALL_THREADS(statement, substr) \
+ do { \
+ class GTestExpectFatalFailureHelper { \
+ public: \
+ static void Execute() { statement; } \
+ }; \
+ ::testing::TestPartResultArray gtest_failures; \
+ ::testing::internal::SingleFailureChecker gtest_checker( \
+ >est_failures, ::testing::TestPartResult::kFatalFailure, (substr)); \
+ { \
+ ::testing::ScopedFakeTestPartResultReporter gtest_reporter( \
+ ::testing::ScopedFakeTestPartResultReporter::INTERCEPT_ALL_THREADS, \
+ >est_failures); \
+ GTestExpectFatalFailureHelper::Execute(); \
+ } \
} while (::testing::internal::AlwaysFalse())
// A macro for testing Google Test assertions or code that's expected to
-// generate Google Test non-fatal failures. It asserts that the given
-// statement will cause exactly one non-fatal Google Test failure with 'substr'
-// being part of the failure message.
+// generate Google Test non-fatal failures (e.g. a failure from an EXPECT_EQ,
+// but not from an ASSERT_EQ). It asserts that the given statement will cause
+// exactly one non-fatal Google Test failure with 'substr' being part of the
+// failure message.
//
// There are two different versions of this macro. EXPECT_NONFATAL_FAILURE only
// affects and considers failures generated in the current thread and
@@ -207,32 +212,37 @@
// instead of
// GTEST_SUPPRESS_UNREACHABLE_CODE_WARNING_BELOW_(statement)
// to avoid an MSVC warning on unreachable code.
-#define EXPECT_NONFATAL_FAILURE(statement, substr) \
- do {\
- ::testing::TestPartResultArray gtest_failures;\
- ::testing::internal::SingleFailureChecker gtest_checker(\
+#define EXPECT_NONFATAL_FAILURE(statement, substr) \
+ do { \
+ ::testing::TestPartResultArray gtest_failures; \
+ ::testing::internal::SingleFailureChecker gtest_checker( \
>est_failures, ::testing::TestPartResult::kNonFatalFailure, \
- (substr));\
- {\
- ::testing::ScopedFakeTestPartResultReporter gtest_reporter(\
- ::testing::ScopedFakeTestPartResultReporter:: \
- INTERCEPT_ONLY_CURRENT_THREAD, >est_failures);\
- if (::testing::internal::AlwaysTrue()) { statement; }\
- }\
+ (substr)); \
+ { \
+ ::testing::ScopedFakeTestPartResultReporter gtest_reporter( \
+ ::testing::ScopedFakeTestPartResultReporter:: \
+ INTERCEPT_ONLY_CURRENT_THREAD, \
+ >est_failures); \
+ if (::testing::internal::AlwaysTrue()) { \
+ statement; \
+ } \
+ } \
} while (::testing::internal::AlwaysFalse())
-#define EXPECT_NONFATAL_FAILURE_ON_ALL_THREADS(statement, substr) \
- do {\
- ::testing::TestPartResultArray gtest_failures;\
- ::testing::internal::SingleFailureChecker gtest_checker(\
- >est_failures, ::testing::TestPartResult::kNonFatalFailure, \
- (substr));\
- {\
- ::testing::ScopedFakeTestPartResultReporter gtest_reporter(\
+#define EXPECT_NONFATAL_FAILURE_ON_ALL_THREADS(statement, substr) \
+ do { \
+ ::testing::TestPartResultArray gtest_failures; \
+ ::testing::internal::SingleFailureChecker gtest_checker( \
+ >est_failures, ::testing::TestPartResult::kNonFatalFailure, \
+ (substr)); \
+ { \
+ ::testing::ScopedFakeTestPartResultReporter gtest_reporter( \
::testing::ScopedFakeTestPartResultReporter::INTERCEPT_ALL_THREADS, \
- >est_failures);\
- if (::testing::internal::AlwaysTrue()) { statement; }\
- }\
+ >est_failures); \
+ if (::testing::internal::AlwaysTrue()) { \
+ statement; \
+ } \
+ } \
} while (::testing::internal::AlwaysFalse())
#endif // GOOGLETEST_INCLUDE_GTEST_GTEST_SPI_H_
diff --git a/third_party/googletest/src/googletest/include/gtest/gtest-test-part.h b/third_party/googletest/src/googletest/include/gtest/gtest-test-part.h
index 203fdf9..09cc8c3 100644
--- a/third_party/googletest/src/googletest/include/gtest/gtest-test-part.h
+++ b/third_party/googletest/src/googletest/include/gtest/gtest-test-part.h
@@ -26,14 +26,17 @@
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
-// GOOGLETEST_CM0001 DO NOT DELETE
+
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLETEST_INCLUDE_GTEST_GTEST_TEST_PART_H_
#define GOOGLETEST_INCLUDE_GTEST_GTEST_TEST_PART_H_
#include <iosfwd>
#include <vector>
+
#include "gtest/internal/gtest-internal.h"
#include "gtest/internal/gtest-string.h"
@@ -142,7 +145,8 @@
private:
std::vector<TestPartResult> array_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(TestPartResultArray);
+ TestPartResultArray(const TestPartResultArray&) = delete;
+ TestPartResultArray& operator=(const TestPartResultArray&) = delete;
};
// This interface knows how to report a test part result.
@@ -168,11 +172,13 @@
~HasNewFatalFailureHelper() override;
void ReportTestPartResult(const TestPartResult& result) override;
bool has_new_fatal_failure() const { return has_new_fatal_failure_; }
+
private:
bool has_new_fatal_failure_;
TestPartResultReporterInterface* original_reporter_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(HasNewFatalFailureHelper);
+ HasNewFatalFailureHelper(const HasNewFatalFailureHelper&) = delete;
+ HasNewFatalFailureHelper& operator=(const HasNewFatalFailureHelper&) = delete;
};
} // namespace internal
diff --git a/third_party/googletest/src/googletest/include/gtest/gtest-typed-test.h b/third_party/googletest/src/googletest/include/gtest/gtest-typed-test.h
index 9fdc6be..bd35a32 100644
--- a/third_party/googletest/src/googletest/include/gtest/gtest-typed-test.h
+++ b/third_party/googletest/src/googletest/include/gtest/gtest-typed-test.h
@@ -27,7 +27,9 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-// GOOGLETEST_CM0001 DO NOT DELETE
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLETEST_INCLUDE_GTEST_GTEST_TYPED_TEST_H_
#define GOOGLETEST_INCLUDE_GTEST_GTEST_TYPED_TEST_H_
@@ -190,7 +192,7 @@
typedef ::testing::internal::GenerateTypeList<Types>::type \
GTEST_TYPE_PARAMS_(CaseName); \
typedef ::testing::internal::NameGeneratorSelector<__VA_ARGS__>::type \
- GTEST_NAME_GENERATOR_(CaseName)
+ GTEST_NAME_GENERATOR_(CaseName)
#define TYPED_TEST(CaseName, TestName) \
static_assert(sizeof(GTEST_STRINGIFY_(TestName)) > 1, \
@@ -256,7 +258,7 @@
// #included in multiple translation units linked together.
#define TYPED_TEST_SUITE_P(SuiteName) \
static ::testing::internal::TypedTestSuitePState \
- GTEST_TYPED_TEST_SUITE_P_STATE_(SuiteName)
+ GTEST_TYPED_TEST_SUITE_P_STATE_(SuiteName)
// Legacy API is deprecated but still available
#ifndef GTEST_REMOVE_LEGACY_TEST_CASEAPI_
@@ -301,21 +303,21 @@
REGISTER_TYPED_TEST_SUITE_P
#endif // GTEST_REMOVE_LEGACY_TEST_CASEAPI_
-#define INSTANTIATE_TYPED_TEST_SUITE_P(Prefix, SuiteName, Types, ...) \
- static_assert(sizeof(GTEST_STRINGIFY_(Prefix)) > 1, \
- "test-suit-prefix must not be empty"); \
- static bool gtest_##Prefix##_##SuiteName GTEST_ATTRIBUTE_UNUSED_ = \
- ::testing::internal::TypeParameterizedTestSuite< \
- SuiteName, GTEST_SUITE_NAMESPACE_(SuiteName)::gtest_AllTests_, \
- ::testing::internal::GenerateTypeList<Types>::type>:: \
- Register(GTEST_STRINGIFY_(Prefix), \
- ::testing::internal::CodeLocation(__FILE__, __LINE__), \
- >EST_TYPED_TEST_SUITE_P_STATE_(SuiteName), \
- GTEST_STRINGIFY_(SuiteName), \
- GTEST_REGISTERED_TEST_NAMES_(SuiteName), \
- ::testing::internal::GenerateNames< \
- ::testing::internal::NameGeneratorSelector< \
- __VA_ARGS__>::type, \
+#define INSTANTIATE_TYPED_TEST_SUITE_P(Prefix, SuiteName, Types, ...) \
+ static_assert(sizeof(GTEST_STRINGIFY_(Prefix)) > 1, \
+ "test-suit-prefix must not be empty"); \
+ static bool gtest_##Prefix##_##SuiteName GTEST_ATTRIBUTE_UNUSED_ = \
+ ::testing::internal::TypeParameterizedTestSuite< \
+ SuiteName, GTEST_SUITE_NAMESPACE_(SuiteName)::gtest_AllTests_, \
+ ::testing::internal::GenerateTypeList<Types>::type>:: \
+ Register(GTEST_STRINGIFY_(Prefix), \
+ ::testing::internal::CodeLocation(__FILE__, __LINE__), \
+ >EST_TYPED_TEST_SUITE_P_STATE_(SuiteName), \
+ GTEST_STRINGIFY_(SuiteName), \
+ GTEST_REGISTERED_TEST_NAMES_(SuiteName), \
+ ::testing::internal::GenerateNames< \
+ ::testing::internal::NameGeneratorSelector< \
+ __VA_ARGS__>::type, \
::testing::internal::GenerateTypeList<Types>::type>())
// Legacy API is deprecated but still available
diff --git a/third_party/googletest/src/googletest/include/gtest/gtest.h b/third_party/googletest/src/googletest/include/gtest/gtest.h
index 7a5d057..d19a587 100644
--- a/third_party/googletest/src/googletest/include/gtest/gtest.h
+++ b/third_party/googletest/src/googletest/include/gtest/gtest.h
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
// The Google C++ Testing and Mocking Framework (Google Test)
//
// This header file defines the public API for Google Test. It should be
@@ -47,8 +46,6 @@
// registration from Barthelemy Dagenais' (barthelemy@prologique.com)
// easyUnit framework.
-// GOOGLETEST_CM0001 DO NOT DELETE
-
#ifndef GOOGLETEST_INCLUDE_GTEST_GTEST_H_
#define GOOGLETEST_INCLUDE_GTEST_GTEST_H_
@@ -59,31 +56,22 @@
#include <type_traits>
#include <vector>
-#include "gtest/internal/gtest-internal.h"
-#include "gtest/internal/gtest-string.h"
+#include "gtest/gtest-assertion-result.h"
#include "gtest/gtest-death-test.h"
#include "gtest/gtest-matchers.h"
#include "gtest/gtest-message.h"
#include "gtest/gtest-param-test.h"
#include "gtest/gtest-printers.h"
-#include "gtest/gtest_prod.h"
#include "gtest/gtest-test-part.h"
#include "gtest/gtest-typed-test.h"
+#include "gtest/gtest_pred_impl.h"
+#include "gtest/gtest_prod.h"
+#include "gtest/internal/gtest-internal.h"
+#include "gtest/internal/gtest-string.h"
GTEST_DISABLE_MSC_WARNINGS_PUSH_(4251 \
/* class A needs to have dll-interface to be used by clients of class B */)
-namespace testing {
-
-// Silence C4100 (unreferenced formal parameter) and 4805
-// unsafe mix of type 'const int' and type 'const bool'
-#ifdef _MSC_VER
-# pragma warning(push)
-# pragma warning(disable:4805)
-# pragma warning(disable:4100)
-#endif
-
-
// Declares the flags.
// This flag temporary enables the disabled tests.
@@ -138,6 +126,12 @@
// is 1. If the value is -1 the tests are repeating forever.
GTEST_DECLARE_int32_(repeat);
+// This flag controls whether Google Test Environments are recreated for each
+// repeat of the tests. The default value is true. If set to false the global
+// test Environment objects are only set up once, for the first iteration, and
+// only torn down once, for the last.
+GTEST_DECLARE_bool_(recreate_environments_when_repeating);
+
// This flag controls whether Google Test includes Google Test internal
// stack frames in failure stack traces.
GTEST_DECLARE_bool_(show_internal_stack_frames);
@@ -163,6 +157,16 @@
GTEST_DECLARE_string_(flagfile);
#endif // GTEST_USE_OWN_FLAGFILE_FLAG_
+namespace testing {
+
+// Silence C4100 (unreferenced formal parameter) and 4805
+// unsafe mix of type 'const int' and type 'const bool'
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4805)
+#pragma warning(disable : 4100)
+#endif
+
// The upper limit for valid stack trace depths.
const int kMaxStackTraceDepth = 100;
@@ -201,193 +205,6 @@
class TestInfo;
class UnitTest;
-// A class for indicating whether an assertion was successful. When
-// the assertion wasn't successful, the AssertionResult object
-// remembers a non-empty message that describes how it failed.
-//
-// To create an instance of this class, use one of the factory functions
-// (AssertionSuccess() and AssertionFailure()).
-//
-// This class is useful for two purposes:
-// 1. Defining predicate functions to be used with Boolean test assertions
-// EXPECT_TRUE/EXPECT_FALSE and their ASSERT_ counterparts
-// 2. Defining predicate-format functions to be
-// used with predicate assertions (ASSERT_PRED_FORMAT*, etc).
-//
-// For example, if you define IsEven predicate:
-//
-// testing::AssertionResult IsEven(int n) {
-// if ((n % 2) == 0)
-// return testing::AssertionSuccess();
-// else
-// return testing::AssertionFailure() << n << " is odd";
-// }
-//
-// Then the failed expectation EXPECT_TRUE(IsEven(Fib(5)))
-// will print the message
-//
-// Value of: IsEven(Fib(5))
-// Actual: false (5 is odd)
-// Expected: true
-//
-// instead of a more opaque
-//
-// Value of: IsEven(Fib(5))
-// Actual: false
-// Expected: true
-//
-// in case IsEven is a simple Boolean predicate.
-//
-// If you expect your predicate to be reused and want to support informative
-// messages in EXPECT_FALSE and ASSERT_FALSE (negative assertions show up
-// about half as often as positive ones in our tests), supply messages for
-// both success and failure cases:
-//
-// testing::AssertionResult IsEven(int n) {
-// if ((n % 2) == 0)
-// return testing::AssertionSuccess() << n << " is even";
-// else
-// return testing::AssertionFailure() << n << " is odd";
-// }
-//
-// Then a statement EXPECT_FALSE(IsEven(Fib(6))) will print
-//
-// Value of: IsEven(Fib(6))
-// Actual: true (8 is even)
-// Expected: false
-//
-// NB: Predicates that support negative Boolean assertions have reduced
-// performance in positive ones so be careful not to use them in tests
-// that have lots (tens of thousands) of positive Boolean assertions.
-//
-// To use this class with EXPECT_PRED_FORMAT assertions such as:
-//
-// // Verifies that Foo() returns an even number.
-// EXPECT_PRED_FORMAT1(IsEven, Foo());
-//
-// you need to define:
-//
-// testing::AssertionResult IsEven(const char* expr, int n) {
-// if ((n % 2) == 0)
-// return testing::AssertionSuccess();
-// else
-// return testing::AssertionFailure()
-// << "Expected: " << expr << " is even\n Actual: it's " << n;
-// }
-//
-// If Foo() returns 5, you will see the following message:
-//
-// Expected: Foo() is even
-// Actual: it's 5
-//
-class GTEST_API_ AssertionResult {
- public:
- // Copy constructor.
- // Used in EXPECT_TRUE/FALSE(assertion_result).
- AssertionResult(const AssertionResult& other);
-
-// C4800 is a level 3 warning in Visual Studio 2015 and earlier.
-// This warning is not emitted in Visual Studio 2017.
-// This warning is off by default starting in Visual Studio 2019 but can be
-// enabled with command-line options.
-#if defined(_MSC_VER) && (_MSC_VER < 1910 || _MSC_VER >= 1920)
- GTEST_DISABLE_MSC_WARNINGS_PUSH_(4800 /* forcing value to bool */)
-#endif
-
- // Used in the EXPECT_TRUE/FALSE(bool_expression).
- //
- // T must be contextually convertible to bool.
- //
- // The second parameter prevents this overload from being considered if
- // the argument is implicitly convertible to AssertionResult. In that case
- // we want AssertionResult's copy constructor to be used.
- template <typename T>
- explicit AssertionResult(
- const T& success,
- typename std::enable_if<
- !std::is_convertible<T, AssertionResult>::value>::type*
- /*enabler*/
- = nullptr)
- : success_(success) {}
-
-#if defined(_MSC_VER) && (_MSC_VER < 1910 || _MSC_VER >= 1920)
- GTEST_DISABLE_MSC_WARNINGS_POP_()
-#endif
-
- // Assignment operator.
- AssertionResult& operator=(AssertionResult other) {
- swap(other);
- return *this;
- }
-
- // Returns true if and only if the assertion succeeded.
- operator bool() const { return success_; } // NOLINT
-
- // Returns the assertion's negation. Used with EXPECT/ASSERT_FALSE.
- AssertionResult operator!() const;
-
- // Returns the text streamed into this AssertionResult. Test assertions
- // use it when they fail (i.e., the predicate's outcome doesn't match the
- // assertion's expectation). When nothing has been streamed into the
- // object, returns an empty string.
- const char* message() const {
- return message_.get() != nullptr ? message_->c_str() : "";
- }
- // Deprecated; please use message() instead.
- const char* failure_message() const { return message(); }
-
- // Streams a custom failure message into this object.
- template <typename T> AssertionResult& operator<<(const T& value) {
- AppendMessage(Message() << value);
- return *this;
- }
-
- // Allows streaming basic output manipulators such as endl or flush into
- // this object.
- AssertionResult& operator<<(
- ::std::ostream& (*basic_manipulator)(::std::ostream& stream)) {
- AppendMessage(Message() << basic_manipulator);
- return *this;
- }
-
- private:
- // Appends the contents of message to message_.
- void AppendMessage(const Message& a_message) {
- if (message_.get() == nullptr) message_.reset(new ::std::string);
- message_->append(a_message.GetString().c_str());
- }
-
- // Swap the contents of this AssertionResult with other.
- void swap(AssertionResult& other);
-
- // Stores result of the assertion predicate.
- bool success_;
- // Stores the message describing the condition in case the expectation
- // construct is not satisfied with the predicate's outcome.
- // Referenced via a pointer to avoid taking too much stack frame space
- // with test assertions.
- std::unique_ptr< ::std::string> message_;
-};
-
-// Makes a successful assertion result.
-GTEST_API_ AssertionResult AssertionSuccess();
-
-// Makes a failed assertion result.
-GTEST_API_ AssertionResult AssertionFailure();
-
-// Makes a failed assertion result with the given failure message.
-// Deprecated; use AssertionFailure() << msg.
-GTEST_API_ AssertionResult AssertionFailure(const Message& msg);
-
-} // namespace testing
-
-// Includes the auto-generated header that implements a family of generic
-// predicate assertion macros. This include comes late because it relies on
-// APIs declared above.
-#include "gtest/gtest_pred_impl.h"
-
-namespace testing {
-
// The abstract class that all tests inherit from.
//
// In Google Test, a unit test program contains one or many TestSuites, and
@@ -522,7 +339,8 @@
virtual Setup_should_be_spelled_SetUp* Setup() { return nullptr; }
// We disallow copying Tests.
- GTEST_DISALLOW_COPY_AND_ASSIGN_(Test);
+ Test(const Test&) = delete;
+ Test& operator=(const Test&) = delete;
};
typedef internal::TimeInMillis TimeInMillis;
@@ -536,24 +354,17 @@
// C'tor. TestProperty does NOT have a default constructor.
// Always use this constructor (with parameters) to create a
// TestProperty object.
- TestProperty(const std::string& a_key, const std::string& a_value) :
- key_(a_key), value_(a_value) {
- }
+ TestProperty(const std::string& a_key, const std::string& a_value)
+ : key_(a_key), value_(a_value) {}
// Gets the user supplied key.
- const char* key() const {
- return key_.c_str();
- }
+ const char* key() const { return key_.c_str(); }
// Gets the user supplied value.
- const char* value() const {
- return value_.c_str();
- }
+ const char* value() const { return value_.c_str(); }
// Sets a new value, overriding the one supplied in the constructor.
- void SetValue(const std::string& new_value) {
- value_ = new_value;
- }
+ void SetValue(const std::string& new_value) { value_ = new_value; }
private:
// The key supplied by the user.
@@ -687,7 +498,8 @@
TimeInMillis elapsed_time_;
// We disallow copying TestResult.
- GTEST_DISALLOW_COPY_AND_ASSIGN_(TestResult);
+ TestResult(const TestResult&) = delete;
+ TestResult& operator=(const TestResult&) = delete;
}; // class TestResult
// A TestInfo object stores the following information about a test:
@@ -811,8 +623,8 @@
}
// These fields are immutable properties of the test.
- const std::string test_suite_name_; // test suite name
- const std::string name_; // Test name
+ const std::string test_suite_name_; // test suite name
+ const std::string name_; // Test name
// Name of the parameter type, or NULL if this is not a typed or a
// type-parameterized test.
const std::unique_ptr<const ::std::string> type_param_;
@@ -833,7 +645,8 @@
// test for the second time.
TestResult result_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(TestInfo);
+ TestInfo(const TestInfo&) = delete;
+ TestInfo& operator=(const TestInfo&) = delete;
};
// A test suite, which consists of a vector of TestInfos.
@@ -941,7 +754,7 @@
// Adds a TestInfo to this test suite. Will delete the TestInfo upon
// destruction of the TestSuite object.
- void AddTestInfo(TestInfo * test_info);
+ void AddTestInfo(TestInfo* test_info);
// Clears the results of all tests in this test suite.
void ClearResult();
@@ -1042,7 +855,8 @@
TestResult ad_hoc_test_result_;
// We disallow copying TestSuites.
- GTEST_DISALLOW_COPY_AND_ASSIGN_(TestSuite);
+ TestSuite(const TestSuite&) = delete;
+ TestSuite& operator=(const TestSuite&) = delete;
};
// An Environment object is capable of setting up and tearing down an
@@ -1069,6 +883,7 @@
// Override this to define how to tear down the environment.
virtual void TearDown() {}
+
private:
// If you see an error about overriding the following function or
// about it being private, you have mis-spelled SetUp() as Setup().
@@ -1120,6 +935,9 @@
// Fired before the test starts.
virtual void OnTestStart(const TestInfo& test_info) = 0;
+ // Fired when a test is disabled
+ virtual void OnTestDisabled(const TestInfo& /*test_info*/) {}
+
// Fired after a failed assertion or a SUCCEED() invocation.
// If you want to throw an exception from this function to skip to the next
// TEST, it must be AssertionException defined above, or inherited from it.
@@ -1143,8 +961,7 @@
virtual void OnEnvironmentsTearDownEnd(const UnitTest& unit_test) = 0;
// Fired after each iteration of tests finishes.
- virtual void OnTestIterationEnd(const UnitTest& unit_test,
- int iteration) = 0;
+ virtual void OnTestIterationEnd(const UnitTest& unit_test, int iteration) = 0;
// Fired after all test activities have ended.
virtual void OnTestProgramEnd(const UnitTest& unit_test) = 0;
@@ -1169,6 +986,7 @@
#endif // GTEST_REMOVE_LEGACY_TEST_CASEAPI_
void OnTestStart(const TestInfo& /*test_info*/) override {}
+ void OnTestDisabled(const TestInfo& /*test_info*/) override {}
void OnTestPartResult(const TestPartResult& /*test_part_result*/) override {}
void OnTestEnd(const TestInfo& /*test_info*/) override {}
void OnTestSuiteEnd(const TestSuite& /*test_suite*/) override {}
@@ -1258,7 +1076,8 @@
TestEventListener* default_xml_generator_;
// We disallow copying TestEventListeners.
- GTEST_DISALLOW_COPY_AND_ASSIGN_(TestEventListeners);
+ TestEventListeners(const TestEventListeners&) = delete;
+ TestEventListeners& operator=(const TestEventListeners&) = delete;
};
// A UnitTest consists of a vector of TestSuites.
@@ -1301,8 +1120,7 @@
// Returns the TestInfo object for the test that's currently running,
// or NULL if no test is running.
- const TestInfo* current_test_info() const
- GTEST_LOCK_EXCLUDED_(mutex_);
+ const TestInfo* current_test_info() const GTEST_LOCK_EXCLUDED_(mutex_);
// Returns the random seed used at the start of the current test run.
int random_seed() const;
@@ -1408,8 +1226,7 @@
// eventually call this to report their results. The user code
// should use the assertion macros instead of calling this directly.
void AddTestPartResult(TestPartResult::Type result_type,
- const char* file_name,
- int line_number,
+ const char* file_name, int line_number,
const std::string& message,
const std::string& os_stack_trace)
GTEST_LOCK_EXCLUDED_(mutex_);
@@ -1440,8 +1257,7 @@
friend std::set<std::string>* internal::GetIgnoredParameterizedTestSuites();
friend internal::UnitTestImpl* internal::GetUnitTestImpl();
friend void internal::ReportFailureInUnknownLocation(
- TestPartResult::Type result_type,
- const std::string& message);
+ TestPartResult::Type result_type, const std::string& message);
// Creates an empty UnitTest.
UnitTest();
@@ -1455,8 +1271,7 @@
GTEST_LOCK_EXCLUDED_(mutex_);
// Pops a trace from the per-thread Google Test trace stack.
- void PopGTestTrace()
- GTEST_LOCK_EXCLUDED_(mutex_);
+ void PopGTestTrace() GTEST_LOCK_EXCLUDED_(mutex_);
// Protects mutable state in *impl_. This is mutable as some const
// methods need to lock it too.
@@ -1469,7 +1284,8 @@
internal::UnitTestImpl* impl_;
// We disallow copying UnitTest.
- GTEST_DISALLOW_COPY_AND_ASSIGN_(UnitTest);
+ UnitTest(const UnitTest&) = delete;
+ UnitTest& operator=(const UnitTest&) = delete;
};
// A convenient wrapper for adding an environment for the test
@@ -1520,13 +1336,11 @@
// when calling EXPECT_* in a tight loop.
template <typename T1, typename T2>
AssertionResult CmpHelperEQFailure(const char* lhs_expression,
- const char* rhs_expression,
- const T1& lhs, const T2& rhs) {
- return EqFailure(lhs_expression,
- rhs_expression,
+ const char* rhs_expression, const T1& lhs,
+ const T2& rhs) {
+ return EqFailure(lhs_expression, rhs_expression,
FormatForComparisonFailureMessage(lhs, rhs),
- FormatForComparisonFailureMessage(rhs, lhs),
- false);
+ FormatForComparisonFailureMessage(rhs, lhs), false);
}
// This block of code defines operator==/!=
@@ -1539,8 +1353,7 @@
// The helper function for {ASSERT|EXPECT}_EQ.
template <typename T1, typename T2>
AssertionResult CmpHelperEQ(const char* lhs_expression,
- const char* rhs_expression,
- const T1& lhs,
+ const char* rhs_expression, const T1& lhs,
const T2& rhs) {
if (lhs == rhs) {
return AssertionSuccess();
@@ -1571,8 +1384,7 @@
// Even though its body looks the same as the above version, we
// cannot merge the two, as it will make anonymous enums unhappy.
static AssertionResult Compare(const char* lhs_expression,
- const char* rhs_expression,
- BiggestInt lhs,
+ const char* rhs_expression, BiggestInt lhs,
BiggestInt rhs) {
return CmpHelperEQ(lhs_expression, rhs_expression, lhs, rhs);
}
@@ -1607,16 +1419,16 @@
//
// INTERNAL IMPLEMENTATION - DO NOT USE IN A USER PROGRAM.
-#define GTEST_IMPL_CMP_HELPER_(op_name, op)\
-template <typename T1, typename T2>\
-AssertionResult CmpHelper##op_name(const char* expr1, const char* expr2, \
- const T1& val1, const T2& val2) {\
- if (val1 op val2) {\
- return AssertionSuccess();\
- } else {\
- return CmpHelperOpFailure(expr1, expr2, val1, val2, #op);\
- }\
-}
+#define GTEST_IMPL_CMP_HELPER_(op_name, op) \
+ template <typename T1, typename T2> \
+ AssertionResult CmpHelper##op_name(const char* expr1, const char* expr2, \
+ const T1& val1, const T2& val2) { \
+ if (val1 op val2) { \
+ return AssertionSuccess(); \
+ } else { \
+ return CmpHelperOpFailure(expr1, expr2, val1, val2, #op); \
+ } \
+ }
// INTERNAL IMPLEMENTATION - DO NOT USE IN A USER PROGRAM.
@@ -1638,49 +1450,42 @@
// INTERNAL IMPLEMENTATION - DO NOT USE IN A USER PROGRAM.
GTEST_API_ AssertionResult CmpHelperSTREQ(const char* s1_expression,
const char* s2_expression,
- const char* s1,
- const char* s2);
+ const char* s1, const char* s2);
// The helper function for {ASSERT|EXPECT}_STRCASEEQ.
//
// INTERNAL IMPLEMENTATION - DO NOT USE IN A USER PROGRAM.
GTEST_API_ AssertionResult CmpHelperSTRCASEEQ(const char* s1_expression,
const char* s2_expression,
- const char* s1,
- const char* s2);
+ const char* s1, const char* s2);
// The helper function for {ASSERT|EXPECT}_STRNE.
//
// INTERNAL IMPLEMENTATION - DO NOT USE IN A USER PROGRAM.
GTEST_API_ AssertionResult CmpHelperSTRNE(const char* s1_expression,
const char* s2_expression,
- const char* s1,
- const char* s2);
+ const char* s1, const char* s2);
// The helper function for {ASSERT|EXPECT}_STRCASENE.
//
// INTERNAL IMPLEMENTATION - DO NOT USE IN A USER PROGRAM.
GTEST_API_ AssertionResult CmpHelperSTRCASENE(const char* s1_expression,
const char* s2_expression,
- const char* s1,
- const char* s2);
-
+ const char* s1, const char* s2);
// Helper function for *_STREQ on wide strings.
//
// INTERNAL IMPLEMENTATION - DO NOT USE IN A USER PROGRAM.
GTEST_API_ AssertionResult CmpHelperSTREQ(const char* s1_expression,
const char* s2_expression,
- const wchar_t* s1,
- const wchar_t* s2);
+ const wchar_t* s1, const wchar_t* s2);
// Helper function for *_STRNE on wide strings.
//
// INTERNAL IMPLEMENTATION - DO NOT USE IN A USER PROGRAM.
GTEST_API_ AssertionResult CmpHelperSTRNE(const char* s1_expression,
const char* s2_expression,
- const wchar_t* s1,
- const wchar_t* s2);
+ const wchar_t* s1, const wchar_t* s2);
} // namespace internal
@@ -1692,32 +1497,40 @@
//
// The {needle,haystack}_expr arguments are the stringified
// expressions that generated the two real arguments.
-GTEST_API_ AssertionResult IsSubstring(
- const char* needle_expr, const char* haystack_expr,
- const char* needle, const char* haystack);
-GTEST_API_ AssertionResult IsSubstring(
- const char* needle_expr, const char* haystack_expr,
- const wchar_t* needle, const wchar_t* haystack);
-GTEST_API_ AssertionResult IsNotSubstring(
- const char* needle_expr, const char* haystack_expr,
- const char* needle, const char* haystack);
-GTEST_API_ AssertionResult IsNotSubstring(
- const char* needle_expr, const char* haystack_expr,
- const wchar_t* needle, const wchar_t* haystack);
-GTEST_API_ AssertionResult IsSubstring(
- const char* needle_expr, const char* haystack_expr,
- const ::std::string& needle, const ::std::string& haystack);
-GTEST_API_ AssertionResult IsNotSubstring(
- const char* needle_expr, const char* haystack_expr,
- const ::std::string& needle, const ::std::string& haystack);
+GTEST_API_ AssertionResult IsSubstring(const char* needle_expr,
+ const char* haystack_expr,
+ const char* needle,
+ const char* haystack);
+GTEST_API_ AssertionResult IsSubstring(const char* needle_expr,
+ const char* haystack_expr,
+ const wchar_t* needle,
+ const wchar_t* haystack);
+GTEST_API_ AssertionResult IsNotSubstring(const char* needle_expr,
+ const char* haystack_expr,
+ const char* needle,
+ const char* haystack);
+GTEST_API_ AssertionResult IsNotSubstring(const char* needle_expr,
+ const char* haystack_expr,
+ const wchar_t* needle,
+ const wchar_t* haystack);
+GTEST_API_ AssertionResult IsSubstring(const char* needle_expr,
+ const char* haystack_expr,
+ const ::std::string& needle,
+ const ::std::string& haystack);
+GTEST_API_ AssertionResult IsNotSubstring(const char* needle_expr,
+ const char* haystack_expr,
+ const ::std::string& needle,
+ const ::std::string& haystack);
#if GTEST_HAS_STD_WSTRING
-GTEST_API_ AssertionResult IsSubstring(
- const char* needle_expr, const char* haystack_expr,
- const ::std::wstring& needle, const ::std::wstring& haystack);
-GTEST_API_ AssertionResult IsNotSubstring(
- const char* needle_expr, const char* haystack_expr,
- const ::std::wstring& needle, const ::std::wstring& haystack);
+GTEST_API_ AssertionResult IsSubstring(const char* needle_expr,
+ const char* haystack_expr,
+ const ::std::wstring& needle,
+ const ::std::wstring& haystack);
+GTEST_API_ AssertionResult IsNotSubstring(const char* needle_expr,
+ const char* haystack_expr,
+ const ::std::wstring& needle,
+ const ::std::wstring& haystack);
#endif // GTEST_HAS_STD_WSTRING
namespace internal {
@@ -1732,8 +1545,7 @@
template <typename RawType>
AssertionResult CmpHelperFloatingPointEQ(const char* lhs_expression,
const char* rhs_expression,
- RawType lhs_value,
- RawType rhs_value) {
+ RawType lhs_value, RawType rhs_value) {
const FloatingPoint<RawType> lhs(lhs_value), rhs(rhs_value);
if (lhs.AlmostEquals(rhs)) {
@@ -1748,10 +1560,8 @@
rhs_ss << std::setprecision(std::numeric_limits<RawType>::digits10 + 2)
<< rhs_value;
- return EqFailure(lhs_expression,
- rhs_expression,
- StringStreamToString(&lhs_ss),
- StringStreamToString(&rhs_ss),
+ return EqFailure(lhs_expression, rhs_expression,
+ StringStreamToString(&lhs_ss), StringStreamToString(&rhs_ss),
false);
}
@@ -1761,8 +1571,7 @@
GTEST_API_ AssertionResult DoubleNearPredFormat(const char* expr1,
const char* expr2,
const char* abs_error_expr,
- double val1,
- double val2,
+ double val1, double val2,
double abs_error);
// INTERNAL IMPLEMENTATION - DO NOT USE IN USER CODE.
@@ -1770,9 +1579,7 @@
class GTEST_API_ AssertHelper {
public:
// Constructor.
- AssertHelper(TestPartResult::Type type,
- const char* file,
- int line,
+ AssertHelper(TestPartResult::Type type, const char* file, int line,
const char* message);
~AssertHelper();
@@ -1786,11 +1593,9 @@
// re-using stack space even for temporary variables, so every EXPECT_EQ
// reserves stack space for another AssertHelper.
struct AssertHelperData {
- AssertHelperData(TestPartResult::Type t,
- const char* srcfile,
- int line_num,
+ AssertHelperData(TestPartResult::Type t, const char* srcfile, int line_num,
const char* msg)
- : type(t), file(srcfile), line(line_num), message(msg) { }
+ : type(t), file(srcfile), line(line_num), message(msg) {}
TestPartResult::Type const type;
const char* const file;
@@ -1798,12 +1603,14 @@
std::string const message;
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(AssertHelperData);
+ AssertHelperData(const AssertHelperData&) = delete;
+ AssertHelperData& operator=(const AssertHelperData&) = delete;
};
AssertHelperData* const data_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(AssertHelper);
+ AssertHelper(const AssertHelper&) = delete;
+ AssertHelper& operator=(const AssertHelper&) = delete;
};
} // namespace internal
@@ -1860,15 +1667,14 @@
private:
// Sets parameter value. The caller is responsible for making sure the value
// remains alive and unchanged throughout the current test.
- static void SetParam(const ParamType* parameter) {
- parameter_ = parameter;
- }
+ static void SetParam(const ParamType* parameter) { parameter_ = parameter; }
// Static value used for accessing parameter during a test lifetime.
static const ParamType* parameter_;
// TestClass must be a subclass of WithParamInterface<T> and Test.
- template <class TestClass> friend class internal::ParameterizedTestFactory;
+ template <class TestClass>
+ friend class internal::ParameterizedTestFactory;
};
template <typename T>
@@ -1878,8 +1684,7 @@
// WithParamInterface, and can just inherit from ::testing::TestWithParam.
template <typename T>
-class TestWithParam : public Test, public WithParamInterface<T> {
-};
+class TestWithParam : public Test, public WithParamInterface<T> {};
// Macros for indicating success/failure in test code.
@@ -1910,7 +1715,7 @@
// Generates a nonfatal failure at the given source file location with
// a generic message.
-#define ADD_FAILURE_AT(file, line) \
+#define ADD_FAILURE_AT(file, line) \
GTEST_MESSAGE_AT_(file, line, "Failed", \
::testing::TestPartResult::kNonFatalFailure)
@@ -1925,7 +1730,7 @@
// Define this macro to 1 to omit the definition of FAIL(), which is a
// generic name and clashes with some other libraries.
#if !GTEST_DONT_DEFINE_FAIL
-# define FAIL() GTEST_FAIL()
+#define FAIL() GTEST_FAIL()
#endif
// Generates a success with a generic message.
@@ -1934,7 +1739,7 @@
// Define this macro to 1 to omit the definition of SUCCEED(), which
// is a generic name and clashes with some other libraries.
#if !GTEST_DONT_DEFINE_SUCCEED
-# define SUCCEED() GTEST_SUCCEED()
+#define SUCCEED() GTEST_SUCCEED()
#endif
// Macros for testing exceptions.
@@ -1962,16 +1767,15 @@
// Boolean assertions. Condition can be either a Boolean expression or an
// AssertionResult. For more information on how to use AssertionResult with
// these macros see comments on that class.
-#define GTEST_EXPECT_TRUE(condition) \
+#define GTEST_EXPECT_TRUE(condition) \
GTEST_TEST_BOOLEAN_(condition, #condition, false, true, \
GTEST_NONFATAL_FAILURE_)
-#define GTEST_EXPECT_FALSE(condition) \
+#define GTEST_EXPECT_FALSE(condition) \
GTEST_TEST_BOOLEAN_(!(condition), #condition, true, false, \
GTEST_NONFATAL_FAILURE_)
#define GTEST_ASSERT_TRUE(condition) \
- GTEST_TEST_BOOLEAN_(condition, #condition, false, true, \
- GTEST_FATAL_FAILURE_)
-#define GTEST_ASSERT_FALSE(condition) \
+ GTEST_TEST_BOOLEAN_(condition, #condition, false, true, GTEST_FATAL_FAILURE_)
+#define GTEST_ASSERT_FALSE(condition) \
GTEST_TEST_BOOLEAN_(!(condition), #condition, true, false, \
GTEST_FATAL_FAILURE_)
@@ -2070,27 +1874,27 @@
// ASSERT_XY(), which clashes with some users' own code.
#if !GTEST_DONT_DEFINE_ASSERT_EQ
-# define ASSERT_EQ(val1, val2) GTEST_ASSERT_EQ(val1, val2)
+#define ASSERT_EQ(val1, val2) GTEST_ASSERT_EQ(val1, val2)
#endif
#if !GTEST_DONT_DEFINE_ASSERT_NE
-# define ASSERT_NE(val1, val2) GTEST_ASSERT_NE(val1, val2)
+#define ASSERT_NE(val1, val2) GTEST_ASSERT_NE(val1, val2)
#endif
#if !GTEST_DONT_DEFINE_ASSERT_LE
-# define ASSERT_LE(val1, val2) GTEST_ASSERT_LE(val1, val2)
+#define ASSERT_LE(val1, val2) GTEST_ASSERT_LE(val1, val2)
#endif
#if !GTEST_DONT_DEFINE_ASSERT_LT
-# define ASSERT_LT(val1, val2) GTEST_ASSERT_LT(val1, val2)
+#define ASSERT_LT(val1, val2) GTEST_ASSERT_LT(val1, val2)
#endif
#if !GTEST_DONT_DEFINE_ASSERT_GE
-# define ASSERT_GE(val1, val2) GTEST_ASSERT_GE(val1, val2)
+#define ASSERT_GE(val1, val2) GTEST_ASSERT_GE(val1, val2)
#endif
#if !GTEST_DONT_DEFINE_ASSERT_GT
-# define ASSERT_GT(val1, val2) GTEST_ASSERT_GT(val1, val2)
+#define ASSERT_GT(val1, val2) GTEST_ASSERT_GT(val1, val2)
#endif
// C-string Comparisons. All tests treat NULL and any non-NULL string
@@ -2115,7 +1919,7 @@
EXPECT_PRED_FORMAT2(::testing::internal::CmpHelperSTRNE, s1, s2)
#define EXPECT_STRCASEEQ(s1, s2) \
EXPECT_PRED_FORMAT2(::testing::internal::CmpHelperSTRCASEEQ, s1, s2)
-#define EXPECT_STRCASENE(s1, s2)\
+#define EXPECT_STRCASENE(s1, s2) \
EXPECT_PRED_FORMAT2(::testing::internal::CmpHelperSTRCASENE, s1, s2)
#define ASSERT_STREQ(s1, s2) \
@@ -2124,7 +1928,7 @@
ASSERT_PRED_FORMAT2(::testing::internal::CmpHelperSTRNE, s1, s2)
#define ASSERT_STRCASEEQ(s1, s2) \
ASSERT_PRED_FORMAT2(::testing::internal::CmpHelperSTRCASEEQ, s1, s2)
-#define ASSERT_STRCASENE(s1, s2)\
+#define ASSERT_STRCASENE(s1, s2) \
ASSERT_PRED_FORMAT2(::testing::internal::CmpHelperSTRCASENE, s1, s2)
// Macros for comparing floating-point numbers.
@@ -2141,29 +1945,29 @@
// FloatingPoint template class in gtest-internal.h if you are
// interested in the implementation details.
-#define EXPECT_FLOAT_EQ(val1, val2)\
+#define EXPECT_FLOAT_EQ(val1, val2) \
EXPECT_PRED_FORMAT2(::testing::internal::CmpHelperFloatingPointEQ<float>, \
val1, val2)
-#define EXPECT_DOUBLE_EQ(val1, val2)\
+#define EXPECT_DOUBLE_EQ(val1, val2) \
EXPECT_PRED_FORMAT2(::testing::internal::CmpHelperFloatingPointEQ<double>, \
val1, val2)
-#define ASSERT_FLOAT_EQ(val1, val2)\
+#define ASSERT_FLOAT_EQ(val1, val2) \
ASSERT_PRED_FORMAT2(::testing::internal::CmpHelperFloatingPointEQ<float>, \
val1, val2)
-#define ASSERT_DOUBLE_EQ(val1, val2)\
+#define ASSERT_DOUBLE_EQ(val1, val2) \
ASSERT_PRED_FORMAT2(::testing::internal::CmpHelperFloatingPointEQ<double>, \
val1, val2)
-#define EXPECT_NEAR(val1, val2, abs_error)\
- EXPECT_PRED_FORMAT3(::testing::internal::DoubleNearPredFormat, \
- val1, val2, abs_error)
+#define EXPECT_NEAR(val1, val2, abs_error) \
+ EXPECT_PRED_FORMAT3(::testing::internal::DoubleNearPredFormat, val1, val2, \
+ abs_error)
-#define ASSERT_NEAR(val1, val2, abs_error)\
- ASSERT_PRED_FORMAT3(::testing::internal::DoubleNearPredFormat, \
- val1, val2, abs_error)
+#define ASSERT_NEAR(val1, val2, abs_error) \
+ ASSERT_PRED_FORMAT3(::testing::internal::DoubleNearPredFormat, val1, val2, \
+ abs_error)
// These predicate format functions work on floating-point values, and
// can be used in {ASSERT|EXPECT}_PRED_FORMAT2*(), e.g.
@@ -2177,7 +1981,6 @@
GTEST_API_ AssertionResult DoubleLE(const char* expr1, const char* expr2,
double val1, double val2);
-
#if GTEST_OS_WINDOWS
// Macros that test for HRESULT failure and success, these are only useful
@@ -2189,17 +1992,17 @@
// expected result and the actual result with both a human-readable
// string representation of the error, if available, as well as the
// hex result code.
-# define EXPECT_HRESULT_SUCCEEDED(expr) \
- EXPECT_PRED_FORMAT1(::testing::internal::IsHRESULTSuccess, (expr))
+#define EXPECT_HRESULT_SUCCEEDED(expr) \
+ EXPECT_PRED_FORMAT1(::testing::internal::IsHRESULTSuccess, (expr))
-# define ASSERT_HRESULT_SUCCEEDED(expr) \
- ASSERT_PRED_FORMAT1(::testing::internal::IsHRESULTSuccess, (expr))
+#define ASSERT_HRESULT_SUCCEEDED(expr) \
+ ASSERT_PRED_FORMAT1(::testing::internal::IsHRESULTSuccess, (expr))
-# define EXPECT_HRESULT_FAILED(expr) \
- EXPECT_PRED_FORMAT1(::testing::internal::IsHRESULTFailure, (expr))
+#define EXPECT_HRESULT_FAILED(expr) \
+ EXPECT_PRED_FORMAT1(::testing::internal::IsHRESULTFailure, (expr))
-# define ASSERT_HRESULT_FAILED(expr) \
- ASSERT_PRED_FORMAT1(::testing::internal::IsHRESULTFailure, (expr))
+#define ASSERT_HRESULT_FAILED(expr) \
+ ASSERT_PRED_FORMAT1(::testing::internal::IsHRESULTFailure, (expr))
#endif // GTEST_OS_WINDOWS
@@ -2214,9 +2017,9 @@
// ASSERT_NO_FATAL_FAILURE(Process()) << "Process() failed";
//
#define ASSERT_NO_FATAL_FAILURE(statement) \
- GTEST_TEST_NO_FATAL_FAILURE_(statement, GTEST_FATAL_FAILURE_)
+ GTEST_TEST_NO_FATAL_FAILURE_(statement, GTEST_FATAL_FAILURE_)
#define EXPECT_NO_FATAL_FAILURE(statement) \
- GTEST_TEST_NO_FATAL_FAILURE_(statement, GTEST_NONFATAL_FAILURE_)
+ GTEST_TEST_NO_FATAL_FAILURE_(statement, GTEST_NONFATAL_FAILURE_)
// Causes a trace (including the given source file path and line number,
// and the given message) to be included in every test failure message generated
@@ -2258,7 +2061,8 @@
private:
void PushTrace(const char* file, int line, std::string message);
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ScopedTrace);
+ ScopedTrace(const ScopedTrace&) = delete;
+ ScopedTrace& operator=(const ScopedTrace&) = delete;
} GTEST_ATTRIBUTE_UNUSED_; // A ScopedTrace object does its job in its
// c'tor and d'tor. Therefore it doesn't
// need to be used otherwise.
@@ -2278,9 +2082,9 @@
// Assuming that each thread maintains its own stack of traces.
// Therefore, a SCOPED_TRACE() would (correctly) only affect the
// assertions in its own thread.
-#define SCOPED_TRACE(message) \
- ::testing::ScopedTrace GTEST_CONCAT_TOKEN_(gtest_trace_, __LINE__)(\
- __FILE__, __LINE__, (message))
+#define SCOPED_TRACE(message) \
+ ::testing::ScopedTrace GTEST_CONCAT_TOKEN_(gtest_trace_, __LINE__)( \
+ __FILE__, __LINE__, (message))
// Compile-time assertion for type equality.
// StaticAssertTypeEq<type1, type2>() compiles if and only if type1 and type2
@@ -2378,20 +2182,19 @@
// EXPECT_EQ(a_.size(), 0);
// EXPECT_EQ(b_.size(), 1);
// }
-//
-// GOOGLETEST_CM0011 DO NOT DELETE
-#if !GTEST_DONT_DEFINE_TEST
-#define TEST_F(test_fixture, test_name)\
+#define GTEST_TEST_F(test_fixture, test_name) \
GTEST_TEST_(test_fixture, test_name, test_fixture, \
::testing::internal::GetTypeId<test_fixture>())
-#endif // !GTEST_DONT_DEFINE_TEST
+#if !GTEST_DONT_DEFINE_TEST_F
+#define TEST_F(test_fixture, test_name) GTEST_TEST_F(test_fixture, test_name)
+#endif
// Returns a path to temporary directory.
// Tries to determine an appropriate directory for the platform.
GTEST_API_ std::string TempDir();
#ifdef _MSC_VER
-# pragma warning(pop)
+#pragma warning(pop)
#endif
// Dynamically registers a test with the framework.
@@ -2445,6 +2248,7 @@
// }
// ...
// int main(int argc, char** argv) {
+// ::testing::InitGoogleTest(&argc, argv);
// std::vector<int> values_to_test = LoadValuesFromConfig();
// RegisterMyTests(values_to_test);
// ...
@@ -2486,9 +2290,7 @@
// namespace and has an all-caps name.
int RUN_ALL_TESTS() GTEST_MUST_USE_RESULT_;
-inline int RUN_ALL_TESTS() {
- return ::testing::UnitTest::GetInstance()->Run();
-}
+inline int RUN_ALL_TESTS() { return ::testing::UnitTest::GetInstance()->Run(); }
GTEST_DISABLE_MSC_WARNINGS_POP_() // 4251
diff --git a/third_party/googletest/src/googletest/include/gtest/gtest_pred_impl.h b/third_party/googletest/src/googletest/include/gtest/gtest_pred_impl.h
index 5029a9b..47a24aa 100644
--- a/third_party/googletest/src/googletest/include/gtest/gtest_pred_impl.h
+++ b/third_party/googletest/src/googletest/include/gtest/gtest_pred_impl.h
@@ -26,17 +26,19 @@
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-// This file is AUTOMATICALLY GENERATED on 01/02/2019 by command
-// 'gen_gtest_pred_impl.py 5'. DO NOT EDIT BY HAND!
//
// Implements a family of generic predicate assertion macros.
-// GOOGLETEST_CM0001 DO NOT DELETE
+
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLETEST_INCLUDE_GTEST_GTEST_PRED_IMPL_H_
#define GOOGLETEST_INCLUDE_GTEST_GTEST_PRED_IMPL_H_
-#include "gtest/gtest.h"
+#include "gtest/gtest-assertion-result.h"
+#include "gtest/internal/gtest-internal.h"
+#include "gtest/internal/gtest-port.h"
namespace testing {
@@ -72,22 +74,18 @@
// GTEST_ASSERT_ is the basic statement to which all of the assertions
// in this file reduce. Don't use this in your code.
-#define GTEST_ASSERT_(expression, on_failure) \
- GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
+#define GTEST_ASSERT_(expression, on_failure) \
+ GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
if (const ::testing::AssertionResult gtest_ar = (expression)) \
- ; \
- else \
+ ; \
+ else \
on_failure(gtest_ar.failure_message())
-
// Helper function for implementing {EXPECT|ASSERT}_PRED1. Don't use
// this in your code.
-template <typename Pred,
- typename T1>
-AssertionResult AssertPred1Helper(const char* pred_text,
- const char* e1,
- Pred pred,
- const T1& v1) {
+template <typename Pred, typename T1>
+AssertionResult AssertPred1Helper(const char* pred_text, const char* e1,
+ Pred pred, const T1& v1) {
if (pred(v1)) return AssertionSuccess();
return AssertionFailure()
@@ -98,40 +96,27 @@
// Internal macro for implementing {EXPECT|ASSERT}_PRED_FORMAT1.
// Don't use this in your code.
-#define GTEST_PRED_FORMAT1_(pred_format, v1, on_failure)\
- GTEST_ASSERT_(pred_format(#v1, v1), \
- on_failure)
+#define GTEST_PRED_FORMAT1_(pred_format, v1, on_failure) \
+ GTEST_ASSERT_(pred_format(#v1, v1), on_failure)
// Internal macro for implementing {EXPECT|ASSERT}_PRED1. Don't use
// this in your code.
-#define GTEST_PRED1_(pred, v1, on_failure)\
- GTEST_ASSERT_(::testing::AssertPred1Helper(#pred, \
- #v1, \
- pred, \
- v1), on_failure)
+#define GTEST_PRED1_(pred, v1, on_failure) \
+ GTEST_ASSERT_(::testing::AssertPred1Helper(#pred, #v1, pred, v1), on_failure)
// Unary predicate assertion macros.
#define EXPECT_PRED_FORMAT1(pred_format, v1) \
GTEST_PRED_FORMAT1_(pred_format, v1, GTEST_NONFATAL_FAILURE_)
-#define EXPECT_PRED1(pred, v1) \
- GTEST_PRED1_(pred, v1, GTEST_NONFATAL_FAILURE_)
+#define EXPECT_PRED1(pred, v1) GTEST_PRED1_(pred, v1, GTEST_NONFATAL_FAILURE_)
#define ASSERT_PRED_FORMAT1(pred_format, v1) \
GTEST_PRED_FORMAT1_(pred_format, v1, GTEST_FATAL_FAILURE_)
-#define ASSERT_PRED1(pred, v1) \
- GTEST_PRED1_(pred, v1, GTEST_FATAL_FAILURE_)
-
-
+#define ASSERT_PRED1(pred, v1) GTEST_PRED1_(pred, v1, GTEST_FATAL_FAILURE_)
// Helper function for implementing {EXPECT|ASSERT}_PRED2. Don't use
// this in your code.
-template <typename Pred,
- typename T1,
- typename T2>
-AssertionResult AssertPred2Helper(const char* pred_text,
- const char* e1,
- const char* e2,
- Pred pred,
- const T1& v1,
+template <typename Pred, typename T1, typename T2>
+AssertionResult AssertPred2Helper(const char* pred_text, const char* e1,
+ const char* e2, Pred pred, const T1& v1,
const T2& v2) {
if (pred(v1, v2)) return AssertionSuccess();
@@ -145,19 +130,14 @@
// Internal macro for implementing {EXPECT|ASSERT}_PRED_FORMAT2.
// Don't use this in your code.
-#define GTEST_PRED_FORMAT2_(pred_format, v1, v2, on_failure)\
- GTEST_ASSERT_(pred_format(#v1, #v2, v1, v2), \
- on_failure)
+#define GTEST_PRED_FORMAT2_(pred_format, v1, v2, on_failure) \
+ GTEST_ASSERT_(pred_format(#v1, #v2, v1, v2), on_failure)
// Internal macro for implementing {EXPECT|ASSERT}_PRED2. Don't use
// this in your code.
-#define GTEST_PRED2_(pred, v1, v2, on_failure)\
- GTEST_ASSERT_(::testing::AssertPred2Helper(#pred, \
- #v1, \
- #v2, \
- pred, \
- v1, \
- v2), on_failure)
+#define GTEST_PRED2_(pred, v1, v2, on_failure) \
+ GTEST_ASSERT_(::testing::AssertPred2Helper(#pred, #v1, #v2, pred, v1, v2), \
+ on_failure)
// Binary predicate assertion macros.
#define EXPECT_PRED_FORMAT2(pred_format, v1, v2) \
@@ -169,22 +149,12 @@
#define ASSERT_PRED2(pred, v1, v2) \
GTEST_PRED2_(pred, v1, v2, GTEST_FATAL_FAILURE_)
-
-
// Helper function for implementing {EXPECT|ASSERT}_PRED3. Don't use
// this in your code.
-template <typename Pred,
- typename T1,
- typename T2,
- typename T3>
-AssertionResult AssertPred3Helper(const char* pred_text,
- const char* e1,
- const char* e2,
- const char* e3,
- Pred pred,
- const T1& v1,
- const T2& v2,
- const T3& v3) {
+template <typename Pred, typename T1, typename T2, typename T3>
+AssertionResult AssertPred3Helper(const char* pred_text, const char* e1,
+ const char* e2, const char* e3, Pred pred,
+ const T1& v1, const T2& v2, const T3& v3) {
if (pred(v1, v2, v3)) return AssertionSuccess();
return AssertionFailure()
@@ -198,21 +168,15 @@
// Internal macro for implementing {EXPECT|ASSERT}_PRED_FORMAT3.
// Don't use this in your code.
-#define GTEST_PRED_FORMAT3_(pred_format, v1, v2, v3, on_failure)\
- GTEST_ASSERT_(pred_format(#v1, #v2, #v3, v1, v2, v3), \
- on_failure)
+#define GTEST_PRED_FORMAT3_(pred_format, v1, v2, v3, on_failure) \
+ GTEST_ASSERT_(pred_format(#v1, #v2, #v3, v1, v2, v3), on_failure)
// Internal macro for implementing {EXPECT|ASSERT}_PRED3. Don't use
// this in your code.
-#define GTEST_PRED3_(pred, v1, v2, v3, on_failure)\
- GTEST_ASSERT_(::testing::AssertPred3Helper(#pred, \
- #v1, \
- #v2, \
- #v3, \
- pred, \
- v1, \
- v2, \
- v3), on_failure)
+#define GTEST_PRED3_(pred, v1, v2, v3, on_failure) \
+ GTEST_ASSERT_( \
+ ::testing::AssertPred3Helper(#pred, #v1, #v2, #v3, pred, v1, v2, v3), \
+ on_failure)
// Ternary predicate assertion macros.
#define EXPECT_PRED_FORMAT3(pred_format, v1, v2, v3) \
@@ -224,25 +188,13 @@
#define ASSERT_PRED3(pred, v1, v2, v3) \
GTEST_PRED3_(pred, v1, v2, v3, GTEST_FATAL_FAILURE_)
-
-
// Helper function for implementing {EXPECT|ASSERT}_PRED4. Don't use
// this in your code.
-template <typename Pred,
- typename T1,
- typename T2,
- typename T3,
- typename T4>
-AssertionResult AssertPred4Helper(const char* pred_text,
- const char* e1,
- const char* e2,
- const char* e3,
- const char* e4,
- Pred pred,
- const T1& v1,
- const T2& v2,
- const T3& v3,
- const T4& v4) {
+template <typename Pred, typename T1, typename T2, typename T3, typename T4>
+AssertionResult AssertPred4Helper(const char* pred_text, const char* e1,
+ const char* e2, const char* e3,
+ const char* e4, Pred pred, const T1& v1,
+ const T2& v2, const T3& v3, const T4& v4) {
if (pred(v1, v2, v3, v4)) return AssertionSuccess();
return AssertionFailure()
@@ -257,23 +209,15 @@
// Internal macro for implementing {EXPECT|ASSERT}_PRED_FORMAT4.
// Don't use this in your code.
-#define GTEST_PRED_FORMAT4_(pred_format, v1, v2, v3, v4, on_failure)\
- GTEST_ASSERT_(pred_format(#v1, #v2, #v3, #v4, v1, v2, v3, v4), \
- on_failure)
+#define GTEST_PRED_FORMAT4_(pred_format, v1, v2, v3, v4, on_failure) \
+ GTEST_ASSERT_(pred_format(#v1, #v2, #v3, #v4, v1, v2, v3, v4), on_failure)
// Internal macro for implementing {EXPECT|ASSERT}_PRED4. Don't use
// this in your code.
-#define GTEST_PRED4_(pred, v1, v2, v3, v4, on_failure)\
- GTEST_ASSERT_(::testing::AssertPred4Helper(#pred, \
- #v1, \
- #v2, \
- #v3, \
- #v4, \
- pred, \
- v1, \
- v2, \
- v3, \
- v4), on_failure)
+#define GTEST_PRED4_(pred, v1, v2, v3, v4, on_failure) \
+ GTEST_ASSERT_(::testing::AssertPred4Helper(#pred, #v1, #v2, #v3, #v4, pred, \
+ v1, v2, v3, v4), \
+ on_failure)
// 4-ary predicate assertion macros.
#define EXPECT_PRED_FORMAT4(pred_format, v1, v2, v3, v4) \
@@ -285,28 +229,15 @@
#define ASSERT_PRED4(pred, v1, v2, v3, v4) \
GTEST_PRED4_(pred, v1, v2, v3, v4, GTEST_FATAL_FAILURE_)
-
-
// Helper function for implementing {EXPECT|ASSERT}_PRED5. Don't use
// this in your code.
-template <typename Pred,
- typename T1,
- typename T2,
- typename T3,
- typename T4,
+template <typename Pred, typename T1, typename T2, typename T3, typename T4,
typename T5>
-AssertionResult AssertPred5Helper(const char* pred_text,
- const char* e1,
- const char* e2,
- const char* e3,
- const char* e4,
- const char* e5,
- Pred pred,
- const T1& v1,
- const T2& v2,
- const T3& v3,
- const T4& v4,
- const T5& v5) {
+AssertionResult AssertPred5Helper(const char* pred_text, const char* e1,
+ const char* e2, const char* e3,
+ const char* e4, const char* e5, Pred pred,
+ const T1& v1, const T2& v2, const T3& v3,
+ const T4& v4, const T5& v5) {
if (pred(v1, v2, v3, v4, v5)) return AssertionSuccess();
return AssertionFailure()
@@ -322,25 +253,16 @@
// Internal macro for implementing {EXPECT|ASSERT}_PRED_FORMAT5.
// Don't use this in your code.
-#define GTEST_PRED_FORMAT5_(pred_format, v1, v2, v3, v4, v5, on_failure)\
+#define GTEST_PRED_FORMAT5_(pred_format, v1, v2, v3, v4, v5, on_failure) \
GTEST_ASSERT_(pred_format(#v1, #v2, #v3, #v4, #v5, v1, v2, v3, v4, v5), \
on_failure)
// Internal macro for implementing {EXPECT|ASSERT}_PRED5. Don't use
// this in your code.
-#define GTEST_PRED5_(pred, v1, v2, v3, v4, v5, on_failure)\
- GTEST_ASSERT_(::testing::AssertPred5Helper(#pred, \
- #v1, \
- #v2, \
- #v3, \
- #v4, \
- #v5, \
- pred, \
- v1, \
- v2, \
- v3, \
- v4, \
- v5), on_failure)
+#define GTEST_PRED5_(pred, v1, v2, v3, v4, v5, on_failure) \
+ GTEST_ASSERT_(::testing::AssertPred5Helper(#pred, #v1, #v2, #v3, #v4, #v5, \
+ pred, v1, v2, v3, v4, v5), \
+ on_failure)
// 5-ary predicate assertion macros.
#define EXPECT_PRED_FORMAT5(pred_format, v1, v2, v3, v4, v5) \
@@ -352,8 +274,6 @@
#define ASSERT_PRED5(pred, v1, v2, v3, v4, v5) \
GTEST_PRED5_(pred, v1, v2, v3, v4, v5, GTEST_FATAL_FAILURE_)
-
-
} // namespace testing
#endif // GOOGLETEST_INCLUDE_GTEST_GTEST_PRED_IMPL_H_
diff --git a/third_party/googletest/src/googletest/include/gtest/gtest_prod.h b/third_party/googletest/src/googletest/include/gtest/gtest_prod.h
index 38b9d85..1f37dc3 100644
--- a/third_party/googletest/src/googletest/include/gtest/gtest_prod.h
+++ b/third_party/googletest/src/googletest/include/gtest/gtest_prod.h
@@ -27,9 +27,8 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
-// Google C++ Testing and Mocking Framework definitions useful in production code.
-// GOOGLETEST_CM0003 DO NOT DELETE
+// Google C++ Testing and Mocking Framework definitions useful in production
+// code.
#ifndef GOOGLETEST_INCLUDE_GTEST_GTEST_PROD_H_
#define GOOGLETEST_INCLUDE_GTEST_GTEST_PROD_H_
@@ -55,7 +54,7 @@
// Note: The test class must be in the same namespace as the class being tested.
// For example, putting MyClassTest in an anonymous namespace will not work.
-#define FRIEND_TEST(test_case_name, test_name)\
-friend class test_case_name##_##test_name##_Test
+#define FRIEND_TEST(test_case_name, test_name) \
+ friend class test_case_name##_##test_name##_Test
#endif // GOOGLETEST_INCLUDE_GTEST_GTEST_PROD_H_
diff --git a/third_party/googletest/src/googletest/include/gtest/internal/custom/README.md b/third_party/googletest/src/googletest/include/gtest/internal/custom/README.md
index ff391fb..cb49e2c 100644
--- a/third_party/googletest/src/googletest/include/gtest/internal/custom/README.md
+++ b/third_party/googletest/src/googletest/include/gtest/internal/custom/README.md
@@ -15,18 +15,6 @@
The following macros can be defined:
-### Flag related macros:
-
-* `GTEST_FLAG(flag_name)`
-* `GTEST_USE_OWN_FLAGFILE_FLAG_` - Define to 0 when the system provides its
- own flagfile flag parsing.
-* `GTEST_DECLARE_bool_(name)`
-* `GTEST_DECLARE_int32_(name)`
-* `GTEST_DECLARE_string_(name)`
-* `GTEST_DEFINE_bool_(name, default_val, doc)`
-* `GTEST_DEFINE_int32_(name, default_val, doc)`
-* `GTEST_DEFINE_string_(name, default_val, doc)`
-
### Logging:
* `GTEST_LOG_(severity)`
diff --git a/third_party/googletest/src/googletest/include/gtest/internal/custom/gtest-port.h b/third_party/googletest/src/googletest/include/gtest/internal/custom/gtest-port.h
index db02881..9b7fb42 100644
--- a/third_party/googletest/src/googletest/include/gtest/internal/custom/gtest-port.h
+++ b/third_party/googletest/src/googletest/include/gtest/internal/custom/gtest-port.h
@@ -34,4 +34,35 @@
#ifndef GOOGLETEST_INCLUDE_GTEST_INTERNAL_CUSTOM_GTEST_PORT_H_
#define GOOGLETEST_INCLUDE_GTEST_INTERNAL_CUSTOM_GTEST_PORT_H_
+// Use a stub Notification class.
+//
+// The built-in Notification class in GoogleTest v1.12.1 uses std::mutex and
+// std::condition_variable. The <mutex> and <condition_variable> headers of
+// mingw32 g++ (GNU 10.0.0) define std::mutex and std::condition_variable only
+// when configured with the posix threads option but don't define them when
+// configured with the win32 threads option. The Notification class is only
+// used in GoogleTest's internal tests. Since we don't build GoogleTest's
+// internal tests, we don't need a working Notification class. Although it's
+// not hard to fix the mingw32 g++ compilation errors by implementing the
+// Notification class using Windows CRITICAL_SECTION and CONDITION_VARIABLE,
+// it's simpler to just use a stub Notification class on all platforms.
+//
+// The default constructor of the stub class is deleted and the declaration of
+// the Notify() method is commented out, so that compilation will fail if any
+// code actually uses the Notification class.
+
+#define GTEST_HAS_NOTIFICATION_ 1
+namespace testing {
+namespace internal {
+class Notification {
+ public:
+ Notification() = delete;
+ Notification(const Notification&) = delete;
+ Notification& operator=(const Notification&) = delete;
+ // void Notify();
+ void WaitForNotification() {}
+};
+} // namespace internal
+} // namespace testing
+
#endif // GOOGLETEST_INCLUDE_GTEST_INTERNAL_CUSTOM_GTEST_PORT_H_
diff --git a/third_party/googletest/src/googletest/include/gtest/internal/gtest-death-test-internal.h b/third_party/googletest/src/googletest/include/gtest/internal/gtest-death-test-internal.h
index 490296d..45580ae 100644
--- a/third_party/googletest/src/googletest/include/gtest/internal/gtest-death-test-internal.h
+++ b/third_party/googletest/src/googletest/include/gtest/internal/gtest-death-test-internal.h
@@ -26,27 +26,31 @@
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
+
// The Google C++ Testing and Mocking Framework (Google Test)
//
// This header file defines internal utilities needed for implementing
// death tests. They are subject to change without notice.
-// GOOGLETEST_CM0001 DO NOT DELETE
+
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_DEATH_TEST_INTERNAL_H_
#define GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_DEATH_TEST_INTERNAL_H_
+#include <stdio.h>
+
+#include <memory>
+
#include "gtest/gtest-matchers.h"
#include "gtest/internal/gtest-internal.h"
-#include <stdio.h>
-#include <memory>
+GTEST_DECLARE_string_(internal_run_death_test);
namespace testing {
namespace internal {
-GTEST_DECLARE_string_(internal_run_death_test);
-
// Names of the flags (needed for parsing Google Test flags).
const char kDeathTestStyleFlag[] = "death_test_style";
const char kDeathTestUseFork[] = "death_test_use_fork";
@@ -83,16 +87,18 @@
static bool Create(const char* statement, Matcher<const std::string&> matcher,
const char* file, int line, DeathTest** test);
DeathTest();
- virtual ~DeathTest() { }
+ virtual ~DeathTest() {}
// A helper class that aborts a death test when it's deleted.
class ReturnSentinel {
public:
- explicit ReturnSentinel(DeathTest* test) : test_(test) { }
+ explicit ReturnSentinel(DeathTest* test) : test_(test) {}
~ReturnSentinel() { test_->Abort(TEST_ENCOUNTERED_RETURN_STATEMENT); }
+
private:
DeathTest* const test_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ReturnSentinel);
+ ReturnSentinel(const ReturnSentinel&) = delete;
+ ReturnSentinel& operator=(const ReturnSentinel&) = delete;
} GTEST_ATTRIBUTE_UNUSED_;
// An enumeration of possible roles that may be taken when a death
@@ -137,7 +143,8 @@
// A string containing a description of the outcome of the last death test.
static std::string last_death_test_message_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(DeathTest);
+ DeathTest(const DeathTest&) = delete;
+ DeathTest& operator=(const DeathTest&) = delete;
};
GTEST_DISABLE_MSC_WARNINGS_POP_() // 4251
@@ -145,7 +152,7 @@
// Factory interface for death tests. May be mocked out for testing.
class DeathTestFactory {
public:
- virtual ~DeathTestFactory() { }
+ virtual ~DeathTestFactory() {}
virtual bool Create(const char* statement,
Matcher<const std::string&> matcher, const char* file,
int line, DeathTest** test) = 0;
@@ -186,28 +193,28 @@
// Traps C++ exceptions escaping statement and reports them as test
// failures. Note that trapping SEH exceptions is not implemented here.
-# if GTEST_HAS_EXCEPTIONS
-# define GTEST_EXECUTE_DEATH_TEST_STATEMENT_(statement, death_test) \
- try { \
- GTEST_SUPPRESS_UNREACHABLE_CODE_WARNING_BELOW_(statement); \
- } catch (const ::std::exception& gtest_exception) { \
- fprintf(\
- stderr, \
- "\n%s: Caught std::exception-derived exception escaping the " \
- "death test statement. Exception message: %s\n", \
+#if GTEST_HAS_EXCEPTIONS
+#define GTEST_EXECUTE_DEATH_TEST_STATEMENT_(statement, death_test) \
+ try { \
+ GTEST_SUPPRESS_UNREACHABLE_CODE_WARNING_BELOW_(statement); \
+ } catch (const ::std::exception& gtest_exception) { \
+ fprintf( \
+ stderr, \
+ "\n%s: Caught std::exception-derived exception escaping the " \
+ "death test statement. Exception message: %s\n", \
::testing::internal::FormatFileLocation(__FILE__, __LINE__).c_str(), \
- gtest_exception.what()); \
- fflush(stderr); \
+ gtest_exception.what()); \
+ fflush(stderr); \
death_test->Abort(::testing::internal::DeathTest::TEST_THREW_EXCEPTION); \
- } catch (...) { \
+ } catch (...) { \
death_test->Abort(::testing::internal::DeathTest::TEST_THREW_EXCEPTION); \
}
-# else
-# define GTEST_EXECUTE_DEATH_TEST_STATEMENT_(statement, death_test) \
+#else
+#define GTEST_EXECUTE_DEATH_TEST_STATEMENT_(statement, death_test) \
GTEST_SUPPRESS_UNREACHABLE_CODE_WARNING_BELOW_(statement)
-# endif
+#endif
// This macro is for implementing ASSERT_DEATH*, EXPECT_DEATH*,
// ASSERT_EXIT*, and EXPECT_EXIT*.
@@ -236,8 +243,6 @@
gtest_dt->Abort(::testing::internal::DeathTest::TEST_DID_NOT_DIE); \
break; \
} \
- default: \
- break; \
} \
} \
} else \
@@ -265,16 +270,12 @@
// RUN_ALL_TESTS was called.
class InternalRunDeathTestFlag {
public:
- InternalRunDeathTestFlag(const std::string& a_file,
- int a_line,
- int an_index,
+ InternalRunDeathTestFlag(const std::string& a_file, int a_line, int an_index,
int a_write_fd)
- : file_(a_file), line_(a_line), index_(an_index),
- write_fd_(a_write_fd) {}
+ : file_(a_file), line_(a_line), index_(an_index), write_fd_(a_write_fd) {}
~InternalRunDeathTestFlag() {
- if (write_fd_ >= 0)
- posix::Close(write_fd_);
+ if (write_fd_ >= 0) posix::Close(write_fd_);
}
const std::string& file() const { return file_; }
@@ -288,7 +289,8 @@
int index_;
int write_fd_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(InternalRunDeathTestFlag);
+ InternalRunDeathTestFlag(const InternalRunDeathTestFlag&) = delete;
+ InternalRunDeathTestFlag& operator=(const InternalRunDeathTestFlag&) = delete;
};
// Returns a newly created InternalRunDeathTestFlag object with fields
diff --git a/third_party/googletest/src/googletest/include/gtest/internal/gtest-filepath.h b/third_party/googletest/src/googletest/include/gtest/internal/gtest-filepath.h
index 0c033ab..a2a60a9 100644
--- a/third_party/googletest/src/googletest/include/gtest/internal/gtest-filepath.h
+++ b/third_party/googletest/src/googletest/include/gtest/internal/gtest-filepath.h
@@ -26,7 +26,7 @@
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
+
// Google Test filepath utilities
//
// This header file declares classes and functions used internally by
@@ -35,7 +35,9 @@
// This file is #included in gtest/internal/gtest-internal.h.
// Do not include this header file separately!
-// GOOGLETEST_CM0001 DO NOT DELETE
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_FILEPATH_H_
#define GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_FILEPATH_H_
@@ -61,8 +63,8 @@
class GTEST_API_ FilePath {
public:
- FilePath() : pathname_("") { }
- FilePath(const FilePath& rhs) : pathname_(rhs.pathname_) { }
+ FilePath() : pathname_("") {}
+ FilePath(const FilePath& rhs) : pathname_(rhs.pathname_) {}
explicit FilePath(const std::string& pathname) : pathname_(pathname) {
Normalize();
@@ -73,9 +75,7 @@
return *this;
}
- void Set(const FilePath& rhs) {
- pathname_ = rhs.pathname_;
- }
+ void Set(const FilePath& rhs) { pathname_ = rhs.pathname_; }
const std::string& string() const { return pathname_; }
const char* c_str() const { return pathname_.c_str(); }
@@ -88,8 +88,7 @@
// than zero (e.g., 12), returns "dir/test_12.xml".
// On Windows platform, uses \ as the separator rather than /.
static FilePath MakeFileName(const FilePath& directory,
- const FilePath& base_name,
- int number,
+ const FilePath& base_name, int number,
const char* extension);
// Given directory = "dir", relative_path = "test.xml",
diff --git a/third_party/googletest/src/googletest/include/gtest/internal/gtest-internal.h b/third_party/googletest/src/googletest/include/gtest/internal/gtest-internal.h
index f8cbdbd..9b04e4c 100644
--- a/third_party/googletest/src/googletest/include/gtest/internal/gtest-internal.h
+++ b/third_party/googletest/src/googletest/include/gtest/internal/gtest-internal.h
@@ -26,13 +26,15 @@
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
+
// The Google C++ Testing and Mocking Framework (Google Test)
//
// This header file declares functions and macros used internally by
// Google Test. They are subject to change without notice.
-// GOOGLETEST_CM0001 DO NOT DELETE
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_INTERNAL_H_
#define GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_INTERNAL_H_
@@ -40,19 +42,20 @@
#include "gtest/internal/gtest-port.h"
#if GTEST_OS_LINUX
-# include <stdlib.h>
-# include <sys/types.h>
-# include <sys/wait.h>
-# include <unistd.h>
+#include <stdlib.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+#include <unistd.h>
#endif // GTEST_OS_LINUX
#if GTEST_HAS_EXCEPTIONS
-# include <stdexcept>
+#include <stdexcept>
#endif
#include <ctype.h>
#include <float.h>
#include <string.h>
+
#include <cstdint>
#include <iomanip>
#include <limits>
@@ -76,7 +79,7 @@
// the current line number. For more details, see
// http://www.parashift.com/c++-faq-lite/misc-technical-issues.html#faq-39.6
#define GTEST_CONCAT_TOKEN_(foo, bar) GTEST_CONCAT_TOKEN_IMPL_(foo, bar)
-#define GTEST_CONCAT_TOKEN_IMPL_(foo, bar) foo ## bar
+#define GTEST_CONCAT_TOKEN_IMPL_(foo, bar) foo##bar
// Stringifies its argument.
// Work around a bug in visual studio which doesn't accept code like this:
@@ -98,21 +101,21 @@
// Forward declarations.
-class AssertionResult; // Result of an assertion.
-class Message; // Represents a failure message.
-class Test; // Represents a test.
-class TestInfo; // Information about a test.
-class TestPartResult; // Result of a test part.
-class UnitTest; // A collection of test suites.
+class AssertionResult; // Result of an assertion.
+class Message; // Represents a failure message.
+class Test; // Represents a test.
+class TestInfo; // Information about a test.
+class TestPartResult; // Result of a test part.
+class UnitTest; // A collection of test suites.
template <typename T>
::std::string PrintToString(const T& value);
namespace internal {
-struct TraceInfo; // Information about a trace point.
-class TestInfoImpl; // Opaque implementation of TestInfo
-class UnitTestImpl; // Opaque implementation of UnitTest
+struct TraceInfo; // Information about a trace point.
+class TestInfoImpl; // Opaque implementation of TestInfo
+class UnitTestImpl; // Opaque implementation of UnitTest
// The text used in failure messages to indicate the start of the
// stack trace.
@@ -121,6 +124,7 @@
// An IgnoredValue object can be implicitly constructed from ANY value.
class IgnoredValue {
struct Sink {};
+
public:
// This constructor template allows any value to be implicitly
// converted to IgnoredValue. The object has no data member and
@@ -136,13 +140,13 @@
};
// Appends the user-supplied message to the Google-Test-generated message.
-GTEST_API_ std::string AppendUserMessage(
- const std::string& gtest_msg, const Message& user_msg);
+GTEST_API_ std::string AppendUserMessage(const std::string& gtest_msg,
+ const Message& user_msg);
#if GTEST_HAS_EXCEPTIONS
-GTEST_DISABLE_MSC_WARNINGS_PUSH_(4275 \
-/* an exported class was derived from a class that was not exported */)
+GTEST_DISABLE_MSC_WARNINGS_PUSH_(
+ 4275 /* an exported class was derived from a class that was not exported */)
// This exception is thrown by (and only by) a failed Google Test
// assertion when GTEST_FLAG(throw_on_failure) is true (if exceptions
@@ -181,14 +185,6 @@
} // namespace edit_distance
-// Calculate the diff between 'left' and 'right' and return it in unified diff
-// format.
-// If not null, stores in 'total_line_count' the total number of lines found
-// in left + right.
-GTEST_API_ std::string DiffStrings(const std::string& left,
- const std::string& right,
- size_t* total_line_count);
-
// Constructs and returns the message for an equality assertion
// (e.g. ASSERT_EQ, EXPECT_STREQ, etc) failure.
//
@@ -212,10 +208,8 @@
// Constructs a failure message for Boolean assertions such as EXPECT_TRUE.
GTEST_API_ std::string GetBoolAssertionFailureMessage(
- const AssertionResult& assertion_result,
- const char* expression_text,
- const char* actual_predicate_value,
- const char* expected_predicate_value);
+ const AssertionResult& assertion_result, const char* expression_text,
+ const char* actual_predicate_value, const char* expected_predicate_value);
// This template class represents an IEEE floating-point number
// (either single-precision or double-precision, depending on the
@@ -256,11 +250,11 @@
// Constants.
// # of bits in a number.
- static const size_t kBitCount = 8*sizeof(RawType);
+ static const size_t kBitCount = 8 * sizeof(RawType);
// # of fraction bits in a number.
static const size_t kFractionBitCount =
- std::numeric_limits<RawType>::digits - 1;
+ std::numeric_limits<RawType>::digits - 1;
// # of exponent bits in a number.
static const size_t kExponentBitCount = kBitCount - 1 - kFractionBitCount;
@@ -269,8 +263,8 @@
static const Bits kSignBitMask = static_cast<Bits>(1) << (kBitCount - 1);
// The mask for the fraction bits.
- static const Bits kFractionBitMask =
- ~static_cast<Bits>(0) >> (kExponentBitCount + 1);
+ static const Bits kFractionBitMask = ~static_cast<Bits>(0) >>
+ (kExponentBitCount + 1);
// The mask for the exponent bits.
static const Bits kExponentBitMask = ~(kSignBitMask | kFractionBitMask);
@@ -309,9 +303,7 @@
}
// Returns the floating-point number that represent positive infinity.
- static RawType Infinity() {
- return ReinterpretBits(kExponentBitMask);
- }
+ static RawType Infinity() { return ReinterpretBits(kExponentBitMask); }
// Returns the maximum representable finite floating-point number.
static RawType Max();
@@ -319,7 +311,7 @@
// Non-static methods
// Returns the bits that represents this number.
- const Bits &bits() const { return u_.bits_; }
+ const Bits& bits() const { return u_.bits_; }
// Returns the exponent bits of this number.
Bits exponent_bits() const { return kExponentBitMask & u_.bits_; }
@@ -348,8 +340,8 @@
// a NAN must return false.
if (is_nan() || rhs.is_nan()) return false;
- return DistanceBetweenSignAndMagnitudeNumbers(u_.bits_, rhs.u_.bits_)
- <= kMaxUlps;
+ return DistanceBetweenSignAndMagnitudeNumbers(u_.bits_, rhs.u_.bits_) <=
+ kMaxUlps;
}
private:
@@ -374,7 +366,7 @@
//
// Read http://en.wikipedia.org/wiki/Signed_number_representations
// for more details on signed number representations.
- static Bits SignAndMagnitudeToBiased(const Bits &sam) {
+ static Bits SignAndMagnitudeToBiased(const Bits& sam) {
if (kSignBitMask & sam) {
// sam represents a negative number.
return ~sam + 1;
@@ -386,8 +378,8 @@
// Given two numbers in the sign-and-magnitude representation,
// returns the distance between them as an unsigned number.
- static Bits DistanceBetweenSignAndMagnitudeNumbers(const Bits &sam1,
- const Bits &sam2) {
+ static Bits DistanceBetweenSignAndMagnitudeNumbers(const Bits& sam1,
+ const Bits& sam2) {
const Bits biased1 = SignAndMagnitudeToBiased(sam1);
const Bits biased2 = SignAndMagnitudeToBiased(sam2);
return (biased1 >= biased2) ? (biased1 - biased2) : (biased2 - biased1);
@@ -399,9 +391,13 @@
// We cannot use std::numeric_limits<T>::max() as it clashes with the max()
// macro defined by <windows.h>.
template <>
-inline float FloatingPoint<float>::Max() { return FLT_MAX; }
+inline float FloatingPoint<float>::Max() {
+ return FLT_MAX;
+}
template <>
-inline double FloatingPoint<double>::Max() { return DBL_MAX; }
+inline double FloatingPoint<double>::Max() {
+ return DBL_MAX;
+}
// Typedefs the instances of the FloatingPoint template class that we
// care to use.
@@ -461,7 +457,8 @@
TestFactoryBase() {}
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(TestFactoryBase);
+ TestFactoryBase(const TestFactoryBase&) = delete;
+ TestFactoryBase& operator=(const TestFactoryBase&) = delete;
};
// This class provides implementation of TeastFactoryBase interface.
@@ -510,11 +507,11 @@
template <typename T>
// Note that SuiteApiResolver inherits from T because
-// SetUpTestSuite()/TearDownTestSuite() could be protected. Ths way
+// SetUpTestSuite()/TearDownTestSuite() could be protected. This way
// SuiteApiResolver can access them.
struct SuiteApiResolver : T {
// testing::Test is only forward declared at this point. So we make it a
- // dependend class for the compiler to be OK with it.
+ // dependent class for the compiler to be OK with it.
using Test =
typename std::conditional<sizeof(T) != 0, ::testing::Test, void>::type;
@@ -654,7 +651,8 @@
if (comma == nullptr) {
return nullptr;
}
- while (IsSpace(*(++comma))) {}
+ while (IsSpace(*(++comma))) {
+ }
return comma;
}
@@ -668,7 +666,7 @@
// Splits a given string on a given delimiter, populating a given
// vector with the fields.
void SplitString(const ::std::string& str, char delimiter,
- ::std::vector< ::std::string>* dest);
+ ::std::vector<::std::string>* dest);
// The default argument to the template below for the case when the user does
// not provide a name generator.
@@ -781,13 +779,13 @@
const std::vector<std::string>& type_names =
GenerateNames<DefaultNameGenerator, Types>()) {
RegisterTypeParameterizedTestSuiteInstantiation(case_name);
- std::string test_name = StripTrailingSpaces(
- GetPrefixUntilComma(test_names));
+ std::string test_name =
+ StripTrailingSpaces(GetPrefixUntilComma(test_names));
if (!state->TestExists(test_name)) {
fprintf(stderr, "Failed to get code location for test %s.%s at %s.",
case_name, test_name.c_str(),
- FormatFileLocation(code_location.file.c_str(),
- code_location.line).c_str());
+ FormatFileLocation(code_location.file.c_str(), code_location.line)
+ .c_str());
fflush(stderr);
posix::Abort();
}
@@ -831,8 +829,8 @@
// For example, if Foo() calls Bar(), which in turn calls
// GetCurrentOsStackTraceExceptTop(..., 1), Foo() will be included in
// the trace but Bar() and GetCurrentOsStackTraceExceptTop() won't.
-GTEST_API_ std::string GetCurrentOsStackTraceExceptTop(
- UnitTest* unit_test, int skip_count);
+GTEST_API_ std::string GetCurrentOsStackTraceExceptTop(UnitTest* unit_test,
+ int skip_count);
// Helpers for suppressing warnings on unreachable code or constant
// condition.
@@ -881,7 +879,8 @@
private:
uint32_t state_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(Random);
+ Random(const Random&) = delete;
+ Random& operator=(const Random&) = delete;
};
// Turns const U&, U&, const U, and U all into U.
@@ -954,7 +953,9 @@
typedef char IsNotContainer;
template <class C>
-IsNotContainer IsContainerTest(long /* dummy */) { return '\0'; }
+IsNotContainer IsContainerTest(long /* dummy */) {
+ return '\0';
+}
// Trait to detect whether a type T is a hash table.
// The heuristic used is that the type contains an inner type `hasher` and does
@@ -1017,11 +1018,13 @@
// This generic version is used when k is 0.
template <typename T, typename U>
-inline bool ArrayEq(const T& lhs, const U& rhs) { return lhs == rhs; }
+inline bool ArrayEq(const T& lhs, const U& rhs) {
+ return lhs == rhs;
+}
// This overload is used when k >= 1.
template <typename T, typename U, size_t N>
-inline bool ArrayEq(const T(&lhs)[N], const U(&rhs)[N]) {
+inline bool ArrayEq(const T (&lhs)[N], const U (&rhs)[N]) {
return internal::ArrayEq(lhs, N, rhs);
}
@@ -1031,8 +1034,7 @@
template <typename T, typename U>
bool ArrayEq(const T* lhs, size_t size, const U* rhs) {
for (size_t i = 0; i != size; i++) {
- if (!internal::ArrayEq(lhs[i], rhs[i]))
- return false;
+ if (!internal::ArrayEq(lhs[i], rhs[i])) return false;
}
return true;
}
@@ -1042,8 +1044,7 @@
template <typename Iter, typename Element>
Iter ArrayAwareFind(Iter begin, Iter end, const Element& elem) {
for (Iter it = begin; it != end; ++it) {
- if (internal::ArrayEq(*it, elem))
- return it;
+ if (internal::ArrayEq(*it, elem)) return it;
}
return end;
}
@@ -1057,11 +1058,13 @@
// This generic version is used when k is 0.
template <typename T, typename U>
-inline void CopyArray(const T& from, U* to) { *to = from; }
+inline void CopyArray(const T& from, U* to) {
+ *to = from;
+}
// This overload is used when k >= 1.
template <typename T, typename U, size_t N>
-inline void CopyArray(const T(&from)[N], U(*to)[N]) {
+inline void CopyArray(const T (&from)[N], U (*to)[N]) {
internal::CopyArray(from, N, *to);
}
@@ -1114,8 +1117,7 @@
}
~NativeArray() {
- if (clone_ != &NativeArray::InitRef)
- delete[] array_;
+ if (clone_ != &NativeArray::InitRef) delete[] array_;
}
// STL-style container methods.
@@ -1123,8 +1125,7 @@
const_iterator begin() const { return array_; }
const_iterator end() const { return array_ + size_; }
bool operator==(const NativeArray& rhs) const {
- return size() == rhs.size() &&
- ArrayEq(begin(), size(), rhs.begin());
+ return size() == rhs.size() && ArrayEq(begin(), size(), rhs.begin());
}
private:
@@ -1335,9 +1336,9 @@
#endif
} // namespace std
-#define GTEST_MESSAGE_AT_(file, line, message, result_type) \
- ::testing::internal::AssertHelper(result_type, file, line, message) \
- = ::testing::Message()
+#define GTEST_MESSAGE_AT_(file, line, message, result_type) \
+ ::testing::internal::AssertHelper(result_type, file, line, message) = \
+ ::testing::Message()
#define GTEST_MESSAGE_(message, result_type) \
GTEST_MESSAGE_AT_(__FILE__, __LINE__, message, result_type)
@@ -1458,103 +1459,112 @@
#endif // GTEST_HAS_EXCEPTIONS
-#define GTEST_TEST_NO_THROW_(statement, fail) \
- GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
- if (::testing::internal::TrueWithString gtest_msg{}) { \
- try { \
- GTEST_SUPPRESS_UNREACHABLE_CODE_WARNING_BELOW_(statement); \
- } \
- GTEST_TEST_NO_THROW_CATCH_STD_EXCEPTION_() \
- catch (...) { \
- gtest_msg.value = "it throws."; \
- goto GTEST_CONCAT_TOKEN_(gtest_label_testnothrow_, __LINE__); \
- } \
- } else \
- GTEST_CONCAT_TOKEN_(gtest_label_testnothrow_, __LINE__): \
- fail(("Expected: " #statement " doesn't throw an exception.\n" \
- " Actual: " + gtest_msg.value).c_str())
+#define GTEST_TEST_NO_THROW_(statement, fail) \
+ GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
+ if (::testing::internal::TrueWithString gtest_msg{}) { \
+ try { \
+ GTEST_SUPPRESS_UNREACHABLE_CODE_WARNING_BELOW_(statement); \
+ } \
+ GTEST_TEST_NO_THROW_CATCH_STD_EXCEPTION_() \
+ catch (...) { \
+ gtest_msg.value = "it throws."; \
+ goto GTEST_CONCAT_TOKEN_(gtest_label_testnothrow_, __LINE__); \
+ } \
+ } else \
+ GTEST_CONCAT_TOKEN_(gtest_label_testnothrow_, __LINE__) \
+ : fail(("Expected: " #statement " doesn't throw an exception.\n" \
+ " Actual: " + \
+ gtest_msg.value) \
+ .c_str())
-#define GTEST_TEST_ANY_THROW_(statement, fail) \
- GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
- if (::testing::internal::AlwaysTrue()) { \
- bool gtest_caught_any = false; \
- try { \
- GTEST_SUPPRESS_UNREACHABLE_CODE_WARNING_BELOW_(statement); \
- } \
- catch (...) { \
- gtest_caught_any = true; \
- } \
- if (!gtest_caught_any) { \
+#define GTEST_TEST_ANY_THROW_(statement, fail) \
+ GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
+ if (::testing::internal::AlwaysTrue()) { \
+ bool gtest_caught_any = false; \
+ try { \
+ GTEST_SUPPRESS_UNREACHABLE_CODE_WARNING_BELOW_(statement); \
+ } catch (...) { \
+ gtest_caught_any = true; \
+ } \
+ if (!gtest_caught_any) { \
goto GTEST_CONCAT_TOKEN_(gtest_label_testanythrow_, __LINE__); \
- } \
- } else \
- GTEST_CONCAT_TOKEN_(gtest_label_testanythrow_, __LINE__): \
- fail("Expected: " #statement " throws an exception.\n" \
- " Actual: it doesn't.")
-
+ } \
+ } else \
+ GTEST_CONCAT_TOKEN_(gtest_label_testanythrow_, __LINE__) \
+ : fail("Expected: " #statement \
+ " throws an exception.\n" \
+ " Actual: it doesn't.")
// Implements Boolean test assertions such as EXPECT_TRUE. expression can be
// either a boolean expression or an AssertionResult. text is a textual
// representation of expression as it was passed into the EXPECT_TRUE.
#define GTEST_TEST_BOOLEAN_(expression, text, actual, expected, fail) \
- GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
- if (const ::testing::AssertionResult gtest_ar_ = \
- ::testing::AssertionResult(expression)) \
- ; \
- else \
- fail(::testing::internal::GetBoolAssertionFailureMessage(\
- gtest_ar_, text, #actual, #expected).c_str())
+ GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
+ if (const ::testing::AssertionResult gtest_ar_ = \
+ ::testing::AssertionResult(expression)) \
+ ; \
+ else \
+ fail(::testing::internal::GetBoolAssertionFailureMessage( \
+ gtest_ar_, text, #actual, #expected) \
+ .c_str())
-#define GTEST_TEST_NO_FATAL_FAILURE_(statement, fail) \
- GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
- if (::testing::internal::AlwaysTrue()) { \
+#define GTEST_TEST_NO_FATAL_FAILURE_(statement, fail) \
+ GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
+ if (::testing::internal::AlwaysTrue()) { \
::testing::internal::HasNewFatalFailureHelper gtest_fatal_failure_checker; \
- GTEST_SUPPRESS_UNREACHABLE_CODE_WARNING_BELOW_(statement); \
- if (gtest_fatal_failure_checker.has_new_fatal_failure()) { \
- goto GTEST_CONCAT_TOKEN_(gtest_label_testnofatal_, __LINE__); \
- } \
- } else \
- GTEST_CONCAT_TOKEN_(gtest_label_testnofatal_, __LINE__): \
- fail("Expected: " #statement " doesn't generate new fatal " \
- "failures in the current thread.\n" \
- " Actual: it does.")
+ GTEST_SUPPRESS_UNREACHABLE_CODE_WARNING_BELOW_(statement); \
+ if (gtest_fatal_failure_checker.has_new_fatal_failure()) { \
+ goto GTEST_CONCAT_TOKEN_(gtest_label_testnofatal_, __LINE__); \
+ } \
+ } else \
+ GTEST_CONCAT_TOKEN_(gtest_label_testnofatal_, __LINE__) \
+ : fail("Expected: " #statement \
+ " doesn't generate new fatal " \
+ "failures in the current thread.\n" \
+ " Actual: it does.")
// Expands to the name of the class that implements the given test.
#define GTEST_TEST_CLASS_NAME_(test_suite_name, test_name) \
test_suite_name##_##test_name##_Test
// Helper macro for defining tests.
-#define GTEST_TEST_(test_suite_name, test_name, parent_class, parent_id) \
- static_assert(sizeof(GTEST_STRINGIFY_(test_suite_name)) > 1, \
- "test_suite_name must not be empty"); \
- static_assert(sizeof(GTEST_STRINGIFY_(test_name)) > 1, \
- "test_name must not be empty"); \
- class GTEST_TEST_CLASS_NAME_(test_suite_name, test_name) \
- : public parent_class { \
- public: \
- GTEST_TEST_CLASS_NAME_(test_suite_name, test_name)() = default; \
- ~GTEST_TEST_CLASS_NAME_(test_suite_name, test_name)() override = default; \
- GTEST_DISALLOW_COPY_AND_ASSIGN_(GTEST_TEST_CLASS_NAME_(test_suite_name, \
- test_name)); \
- GTEST_DISALLOW_MOVE_AND_ASSIGN_(GTEST_TEST_CLASS_NAME_(test_suite_name, \
- test_name)); \
- \
- private: \
- void TestBody() override; \
- static ::testing::TestInfo* const test_info_ GTEST_ATTRIBUTE_UNUSED_; \
- }; \
- \
- ::testing::TestInfo* const GTEST_TEST_CLASS_NAME_(test_suite_name, \
- test_name)::test_info_ = \
- ::testing::internal::MakeAndRegisterTestInfo( \
- #test_suite_name, #test_name, nullptr, nullptr, \
- ::testing::internal::CodeLocation(__FILE__, __LINE__), (parent_id), \
- ::testing::internal::SuiteApiResolver< \
- parent_class>::GetSetUpCaseOrSuite(__FILE__, __LINE__), \
- ::testing::internal::SuiteApiResolver< \
- parent_class>::GetTearDownCaseOrSuite(__FILE__, __LINE__), \
- new ::testing::internal::TestFactoryImpl<GTEST_TEST_CLASS_NAME_( \
- test_suite_name, test_name)>); \
+#define GTEST_TEST_(test_suite_name, test_name, parent_class, parent_id) \
+ static_assert(sizeof(GTEST_STRINGIFY_(test_suite_name)) > 1, \
+ "test_suite_name must not be empty"); \
+ static_assert(sizeof(GTEST_STRINGIFY_(test_name)) > 1, \
+ "test_name must not be empty"); \
+ class GTEST_TEST_CLASS_NAME_(test_suite_name, test_name) \
+ : public parent_class { \
+ public: \
+ GTEST_TEST_CLASS_NAME_(test_suite_name, test_name)() = default; \
+ ~GTEST_TEST_CLASS_NAME_(test_suite_name, test_name)() override = default; \
+ GTEST_TEST_CLASS_NAME_(test_suite_name, test_name) \
+ (const GTEST_TEST_CLASS_NAME_(test_suite_name, test_name) &) = delete; \
+ GTEST_TEST_CLASS_NAME_(test_suite_name, test_name) & operator=( \
+ const GTEST_TEST_CLASS_NAME_(test_suite_name, \
+ test_name) &) = delete; /* NOLINT */ \
+ GTEST_TEST_CLASS_NAME_(test_suite_name, test_name) \
+ (GTEST_TEST_CLASS_NAME_(test_suite_name, test_name) &&) noexcept = delete; \
+ GTEST_TEST_CLASS_NAME_(test_suite_name, test_name) & operator=( \
+ GTEST_TEST_CLASS_NAME_(test_suite_name, \
+ test_name) &&) noexcept = delete; /* NOLINT */ \
+ \
+ private: \
+ void TestBody() override; \
+ static ::testing::TestInfo* const test_info_ GTEST_ATTRIBUTE_UNUSED_; \
+ }; \
+ \
+ ::testing::TestInfo* const GTEST_TEST_CLASS_NAME_(test_suite_name, \
+ test_name)::test_info_ = \
+ ::testing::internal::MakeAndRegisterTestInfo( \
+ #test_suite_name, #test_name, nullptr, nullptr, \
+ ::testing::internal::CodeLocation(__FILE__, __LINE__), (parent_id), \
+ ::testing::internal::SuiteApiResolver< \
+ parent_class>::GetSetUpCaseOrSuite(__FILE__, __LINE__), \
+ ::testing::internal::SuiteApiResolver< \
+ parent_class>::GetTearDownCaseOrSuite(__FILE__, __LINE__), \
+ new ::testing::internal::TestFactoryImpl<GTEST_TEST_CLASS_NAME_( \
+ test_suite_name, test_name)>); \
void GTEST_TEST_CLASS_NAME_(test_suite_name, test_name)::TestBody()
#endif // GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_INTERNAL_H_
diff --git a/third_party/googletest/src/googletest/include/gtest/internal/gtest-param-util.h b/third_party/googletest/src/googletest/include/gtest/internal/gtest-param-util.h
index c2ef6e3..e7af2f9 100644
--- a/third_party/googletest/src/googletest/include/gtest/internal/gtest-param-util.h
+++ b/third_party/googletest/src/googletest/include/gtest/internal/gtest-param-util.h
@@ -27,10 +27,11 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Type and function utilities for implementing parameterized tests.
-// GOOGLETEST_CM0001 DO NOT DELETE
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_PARAM_UTIL_H_
#define GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_PARAM_UTIL_H_
@@ -46,19 +47,18 @@
#include <utility>
#include <vector>
-#include "gtest/internal/gtest-internal.h"
-#include "gtest/internal/gtest-port.h"
#include "gtest/gtest-printers.h"
#include "gtest/gtest-test-part.h"
+#include "gtest/internal/gtest-internal.h"
+#include "gtest/internal/gtest-port.h"
namespace testing {
// Input to a parameterized test name generator, describing a test parameter.
// Consists of the parameter value and the integer parameter index.
template <class ParamType>
struct TestParamInfo {
- TestParamInfo(const ParamType& a_param, size_t an_index) :
- param(a_param),
- index(an_index) {}
+ TestParamInfo(const ParamType& a_param, size_t an_index)
+ : param(a_param), index(an_index) {}
ParamType param;
size_t index;
};
@@ -84,8 +84,10 @@
GTEST_API_ void ReportInvalidTestSuiteType(const char* test_suite_name,
CodeLocation code_location);
-template <typename> class ParamGeneratorInterface;
-template <typename> class ParamGenerator;
+template <typename>
+class ParamGeneratorInterface;
+template <typename>
+class ParamGenerator;
// Interface for iterating over elements provided by an implementation
// of ParamGeneratorInterface<T>.
@@ -129,8 +131,7 @@
// ParamIterator assumes ownership of the impl_ pointer.
ParamIterator(const ParamIterator& other) : impl_(other.impl_->Clone()) {}
ParamIterator& operator=(const ParamIterator& other) {
- if (this != &other)
- impl_.reset(other.impl_->Clone());
+ if (this != &other) impl_.reset(other.impl_->Clone());
return *this;
}
@@ -157,7 +158,7 @@
private:
friend class ParamGenerator<T>;
explicit ParamIterator(ParamIteratorInterface<T>* impl) : impl_(impl) {}
- std::unique_ptr<ParamIteratorInterface<T> > impl_;
+ std::unique_ptr<ParamIteratorInterface<T>> impl_;
};
// ParamGeneratorInterface<T> is the binary interface to access generators
@@ -179,7 +180,7 @@
// This class implements copy initialization semantics and the contained
// ParamGeneratorInterface<T> instance is shared among all copies
// of the original object. This is possible because that instance is immutable.
-template<typename T>
+template <typename T>
class ParamGenerator {
public:
typedef ParamIterator<T> iterator;
@@ -196,7 +197,7 @@
iterator end() const { return iterator(impl_->End()); }
private:
- std::shared_ptr<const ParamGeneratorInterface<T> > impl_;
+ std::shared_ptr<const ParamGeneratorInterface<T>> impl_;
};
// Generates values from a range of two comparable values. Can be used to
@@ -207,8 +208,10 @@
class RangeGenerator : public ParamGeneratorInterface<T> {
public:
RangeGenerator(T begin, T end, IncrementT step)
- : begin_(begin), end_(end),
- step_(step), end_index_(CalculateEndIndex(begin, end, step)) {}
+ : begin_(begin),
+ end_(end),
+ step_(step),
+ end_index_(CalculateEndIndex(begin, end, step)) {}
~RangeGenerator() override {}
ParamIteratorInterface<T>* Begin() const override {
@@ -251,7 +254,9 @@
private:
Iterator(const Iterator& other)
: ParamIteratorInterface<T>(),
- base_(other.base_), value_(other.value_), index_(other.index_),
+ base_(other.base_),
+ value_(other.value_),
+ index_(other.index_),
step_(other.step_) {}
// No implementation - assignment is unsupported.
@@ -263,12 +268,10 @@
const IncrementT step_;
}; // class RangeGenerator::Iterator
- static int CalculateEndIndex(const T& begin,
- const T& end,
+ static int CalculateEndIndex(const T& begin, const T& end,
const IncrementT& step) {
int end_index = 0;
- for (T i = begin; i < end; i = static_cast<T>(i + step))
- end_index++;
+ for (T i = begin; i < end; i = static_cast<T>(i + step)) end_index++;
return end_index;
}
@@ -283,7 +286,6 @@
const int end_index_;
}; // class RangeGenerator
-
// Generates values from a pair of STL-style iterators. Used in the
// ValuesIn() function. The elements are copied from the source range
// since the source can be located on the stack, and the generator
@@ -341,13 +343,13 @@
<< "The program attempted to compare iterators "
<< "from different generators." << std::endl;
return iterator_ ==
- CheckedDowncastToActualType<const Iterator>(&other)->iterator_;
+ CheckedDowncastToActualType<const Iterator>(&other)->iterator_;
}
private:
Iterator(const Iterator& other)
- // The explicit constructor call suppresses a false warning
- // emitted by gcc when supplied with the -Wextra option.
+ // The explicit constructor call suppresses a false warning
+ // emitted by gcc when supplied with the -Wextra option.
: ParamIteratorInterface<T>(),
base_(other.base_),
iterator_(other.iterator_) {}
@@ -394,8 +396,8 @@
class ParameterizedTestFactory : public TestFactoryBase {
public:
typedef typename TestClass::ParamType ParamType;
- explicit ParameterizedTestFactory(ParamType parameter) :
- parameter_(parameter) {}
+ explicit ParameterizedTestFactory(ParamType parameter)
+ : parameter_(parameter) {}
Test* CreateTest() override {
TestClass::SetParam(¶meter_);
return new TestClass();
@@ -404,7 +406,8 @@
private:
const ParamType parameter_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ParameterizedTestFactory);
+ ParameterizedTestFactory(const ParameterizedTestFactory&) = delete;
+ ParameterizedTestFactory& operator=(const ParameterizedTestFactory&) = delete;
};
// INTERNAL IMPLEMENTATION - DO NOT USE IN USER CODE.
@@ -440,7 +443,8 @@
}
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(TestMetaFactory);
+ TestMetaFactory(const TestMetaFactory&) = delete;
+ TestMetaFactory& operator=(const TestMetaFactory&) = delete;
};
// INTERNAL IMPLEMENTATION - DO NOT USE IN USER CODE.
@@ -471,7 +475,10 @@
ParameterizedTestSuiteInfoBase() {}
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ParameterizedTestSuiteInfoBase);
+ ParameterizedTestSuiteInfoBase(const ParameterizedTestSuiteInfoBase&) =
+ delete;
+ ParameterizedTestSuiteInfoBase& operator=(
+ const ParameterizedTestSuiteInfoBase&) = delete;
};
// INTERNAL IMPLEMENTATION - DO NOT USE IN USER CODE.
@@ -547,8 +554,8 @@
test_it != tests_.end(); ++test_it) {
std::shared_ptr<TestInfo> test_info = *test_it;
for (typename InstantiationContainer::iterator gen_it =
- instantiations_.begin(); gen_it != instantiations_.end();
- ++gen_it) {
+ instantiations_.begin();
+ gen_it != instantiations_.end(); ++gen_it) {
const std::string& instantiation_name = gen_it->name;
ParamGenerator<ParamType> generator((*gen_it->generator)());
ParamNameGeneratorFunc* name_func = gen_it->name_func;
@@ -556,7 +563,7 @@
int line = gen_it->line;
std::string test_suite_name;
- if ( !instantiation_name.empty() )
+ if (!instantiation_name.empty())
test_suite_name = instantiation_name + "/";
test_suite_name += test_info->test_suite_base_name;
@@ -569,17 +576,16 @@
Message test_name_stream;
- std::string param_name = name_func(
- TestParamInfo<ParamType>(*param_it, i));
+ std::string param_name =
+ name_func(TestParamInfo<ParamType>(*param_it, i));
GTEST_CHECK_(IsValidParamName(param_name))
<< "Parameterized test name '" << param_name
- << "' is invalid, in " << file
- << " line " << line << std::endl;
+ << "' is invalid, in " << file << " line " << line << std::endl;
GTEST_CHECK_(test_param_names.count(param_name) == 0)
- << "Duplicate parameterized test name '" << param_name
- << "', in " << file << " line " << line << std::endl;
+ << "Duplicate parameterized test name '" << param_name << "', in "
+ << file << " line " << line << std::endl;
test_param_names.insert(param_name);
@@ -596,15 +602,15 @@
SuiteApiResolver<TestSuite>::GetTearDownCaseOrSuite(file, line),
test_info->test_meta_factory->CreateTestFactory(*param_it));
} // for param_it
- } // for gen_it
- } // for test_it
+ } // for gen_it
+ } // for test_it
if (!generated_instantiations) {
// There are no generaotrs, or they all generate nothing ...
InsertSyntheticTestCase(GetTestSuiteName(), code_location_,
!tests_.empty());
}
- } // RegisterTests
+ } // RegisterTests
private:
// LocalTestInfo structure keeps information about a single test registered
@@ -620,42 +626,39 @@
const std::string test_suite_base_name;
const std::string test_base_name;
- const std::unique_ptr<TestMetaFactoryBase<ParamType> > test_meta_factory;
+ const std::unique_ptr<TestMetaFactoryBase<ParamType>> test_meta_factory;
const CodeLocation code_location;
};
- using TestInfoContainer = ::std::vector<std::shared_ptr<TestInfo> >;
+ using TestInfoContainer = ::std::vector<std::shared_ptr<TestInfo>>;
// Records data received from INSTANTIATE_TEST_SUITE_P macros:
// <Instantiation name, Sequence generator creation function,
// Name generator function, Source file, Source line>
struct InstantiationInfo {
- InstantiationInfo(const std::string &name_in,
- GeneratorCreationFunc* generator_in,
- ParamNameGeneratorFunc* name_func_in,
- const char* file_in,
- int line_in)
- : name(name_in),
- generator(generator_in),
- name_func(name_func_in),
- file(file_in),
- line(line_in) {}
+ InstantiationInfo(const std::string& name_in,
+ GeneratorCreationFunc* generator_in,
+ ParamNameGeneratorFunc* name_func_in, const char* file_in,
+ int line_in)
+ : name(name_in),
+ generator(generator_in),
+ name_func(name_func_in),
+ file(file_in),
+ line(line_in) {}
- std::string name;
- GeneratorCreationFunc* generator;
- ParamNameGeneratorFunc* name_func;
- const char* file;
- int line;
+ std::string name;
+ GeneratorCreationFunc* generator;
+ ParamNameGeneratorFunc* name_func;
+ const char* file;
+ int line;
};
typedef ::std::vector<InstantiationInfo> InstantiationContainer;
static bool IsValidParamName(const std::string& name) {
// Check for empty string
- if (name.empty())
- return false;
+ if (name.empty()) return false;
// Check for invalid characters
for (std::string::size_type index = 0; index < name.size(); ++index) {
- if (!IsAlNum(name[index]) && name[index] != '_')
- return false;
+ if (!IsAlNum(name[index]) && name[index] != '_') return false;
}
return true;
@@ -666,7 +669,9 @@
TestInfoContainer tests_;
InstantiationContainer instantiations_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ParameterizedTestSuiteInfo);
+ ParameterizedTestSuiteInfo(const ParameterizedTestSuiteInfo&) = delete;
+ ParameterizedTestSuiteInfo& operator=(const ParameterizedTestSuiteInfo&) =
+ delete;
}; // class ParameterizedTestSuiteInfo
// Legacy API is deprecated but still available
@@ -709,7 +714,7 @@
// type we are looking for, so we downcast it to that type
// without further checks.
typed_test_info = CheckedDowncastToActualType<
- ParameterizedTestSuiteInfo<TestSuite> >(test_suite_info);
+ ParameterizedTestSuiteInfo<TestSuite>>(test_suite_info);
}
break;
}
@@ -741,7 +746,10 @@
TestSuiteInfoContainer test_suite_infos_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ParameterizedTestSuiteRegistry);
+ ParameterizedTestSuiteRegistry(const ParameterizedTestSuiteRegistry&) =
+ delete;
+ ParameterizedTestSuiteRegistry& operator=(
+ const ParameterizedTestSuiteRegistry&) = delete;
};
// Keep track of what type-parameterized test suite are defined and
@@ -836,7 +844,8 @@
: public ParamIteratorInterface<ParamType> {
public:
IteratorImpl(const ParamGeneratorInterface<ParamType>* base,
- const std::tuple<ParamGenerator<T>...>& generators, bool is_end)
+ const std::tuple<ParamGenerator<T>...>& generators,
+ bool is_end)
: base_(base),
begin_(std::get<I>(generators).begin()...),
end_(std::get<I>(generators).end()...),
diff --git a/third_party/googletest/src/googletest/include/gtest/internal/gtest-port-arch.h b/third_party/googletest/src/googletest/include/gtest/internal/gtest-port-arch.h
index dd84591..f025db7 100644
--- a/third_party/googletest/src/googletest/include/gtest/internal/gtest-port-arch.h
+++ b/third_party/googletest/src/googletest/include/gtest/internal/gtest-port-arch.h
@@ -26,7 +26,7 @@
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
+
// The Google C++ Testing and Mocking Framework (Google Test)
//
// This header file defines the GTEST_OS_* macro.
@@ -37,70 +37,72 @@
// Determines the platform on which Google Test is compiled.
#ifdef __CYGWIN__
-# define GTEST_OS_CYGWIN 1
-# elif defined(__MINGW__) || defined(__MINGW32__) || defined(__MINGW64__)
-# define GTEST_OS_WINDOWS_MINGW 1
-# define GTEST_OS_WINDOWS 1
+#define GTEST_OS_CYGWIN 1
+#elif defined(__MINGW__) || defined(__MINGW32__) || defined(__MINGW64__)
+#define GTEST_OS_WINDOWS_MINGW 1
+#define GTEST_OS_WINDOWS 1
#elif defined _WIN32
-# define GTEST_OS_WINDOWS 1
-# ifdef _WIN32_WCE
-# define GTEST_OS_WINDOWS_MOBILE 1
-# elif defined(WINAPI_FAMILY)
-# include <winapifamily.h>
-# if WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_DESKTOP)
-# define GTEST_OS_WINDOWS_DESKTOP 1
-# elif WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_PHONE_APP)
-# define GTEST_OS_WINDOWS_PHONE 1
-# elif WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_APP)
-# define GTEST_OS_WINDOWS_RT 1
-# elif WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_TV_TITLE)
-# define GTEST_OS_WINDOWS_PHONE 1
-# define GTEST_OS_WINDOWS_TV_TITLE 1
-# else
- // WINAPI_FAMILY defined but no known partition matched.
- // Default to desktop.
-# define GTEST_OS_WINDOWS_DESKTOP 1
-# endif
-# else
-# define GTEST_OS_WINDOWS_DESKTOP 1
-# endif // _WIN32_WCE
+#define GTEST_OS_WINDOWS 1
+#ifdef _WIN32_WCE
+#define GTEST_OS_WINDOWS_MOBILE 1
+#elif defined(WINAPI_FAMILY)
+#include <winapifamily.h>
+#if WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_DESKTOP)
+#define GTEST_OS_WINDOWS_DESKTOP 1
+#elif WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_PHONE_APP)
+#define GTEST_OS_WINDOWS_PHONE 1
+#elif WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_APP)
+#define GTEST_OS_WINDOWS_RT 1
+#elif WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_TV_TITLE)
+#define GTEST_OS_WINDOWS_PHONE 1
+#define GTEST_OS_WINDOWS_TV_TITLE 1
+#else
+// WINAPI_FAMILY defined but no known partition matched.
+// Default to desktop.
+#define GTEST_OS_WINDOWS_DESKTOP 1
+#endif
+#else
+#define GTEST_OS_WINDOWS_DESKTOP 1
+#endif // _WIN32_WCE
#elif defined __OS2__
-# define GTEST_OS_OS2 1
+#define GTEST_OS_OS2 1
#elif defined __APPLE__
-# define GTEST_OS_MAC 1
-# include <TargetConditionals.h>
-# if TARGET_OS_IPHONE
-# define GTEST_OS_IOS 1
-# endif
+#define GTEST_OS_MAC 1
+#include <TargetConditionals.h>
+#if TARGET_OS_IPHONE
+#define GTEST_OS_IOS 1
+#endif
#elif defined __DragonFly__
-# define GTEST_OS_DRAGONFLY 1
+#define GTEST_OS_DRAGONFLY 1
#elif defined __FreeBSD__
-# define GTEST_OS_FREEBSD 1
+#define GTEST_OS_FREEBSD 1
#elif defined __Fuchsia__
-# define GTEST_OS_FUCHSIA 1
+#define GTEST_OS_FUCHSIA 1
+#elif defined(__GNU__)
+#define GTEST_OS_GNU_HURD 1
#elif defined(__GLIBC__) && defined(__FreeBSD_kernel__)
-# define GTEST_OS_GNU_KFREEBSD 1
+#define GTEST_OS_GNU_KFREEBSD 1
#elif defined __linux__
-# define GTEST_OS_LINUX 1
-# if defined __ANDROID__
-# define GTEST_OS_LINUX_ANDROID 1
-# endif
+#define GTEST_OS_LINUX 1
+#if defined __ANDROID__
+#define GTEST_OS_LINUX_ANDROID 1
+#endif
#elif defined __MVS__
-# define GTEST_OS_ZOS 1
+#define GTEST_OS_ZOS 1
#elif defined(__sun) && defined(__SVR4)
-# define GTEST_OS_SOLARIS 1
+#define GTEST_OS_SOLARIS 1
#elif defined(_AIX)
-# define GTEST_OS_AIX 1
+#define GTEST_OS_AIX 1
#elif defined(__hpux)
-# define GTEST_OS_HPUX 1
+#define GTEST_OS_HPUX 1
#elif defined __native_client__
-# define GTEST_OS_NACL 1
+#define GTEST_OS_NACL 1
#elif defined __NetBSD__
-# define GTEST_OS_NETBSD 1
+#define GTEST_OS_NETBSD 1
#elif defined __OpenBSD__
-# define GTEST_OS_OPENBSD 1
+#define GTEST_OS_OPENBSD 1
#elif defined __QNX__
-# define GTEST_OS_QNX 1
+#define GTEST_OS_QNX 1
#elif defined(__HAIKU__)
#define GTEST_OS_HAIKU 1
#elif defined ESP8266
diff --git a/third_party/googletest/src/googletest/include/gtest/internal/gtest-port.h b/third_party/googletest/src/googletest/include/gtest/internal/gtest-port.h
index 0953a78..0003d27 100644
--- a/third_party/googletest/src/googletest/include/gtest/internal/gtest-port.h
+++ b/third_party/googletest/src/googletest/include/gtest/internal/gtest-port.h
@@ -26,7 +26,7 @@
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
+
// Low-level types and utilities for porting Google Test to various
// platforms. All macros ending with _ and symbols defined in an
// internal namespace are subject to change without notice. Code
@@ -38,7 +38,9 @@
// files are expected to #include this. Therefore, it cannot #include
// any other Google Test header.
-// GOOGLETEST_CM0001 DO NOT DELETE
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_PORT_H_
#define GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_PORT_H_
@@ -116,6 +118,7 @@
// GTEST_OS_DRAGONFLY - DragonFlyBSD
// GTEST_OS_FREEBSD - FreeBSD
// GTEST_OS_FUCHSIA - Fuchsia
+// GTEST_OS_GNU_HURD - GNU/Hurd
// GTEST_OS_GNU_KFREEBSD - GNU/kFreeBSD
// GTEST_OS_HAIKU - Haiku
// GTEST_OS_HPUX - HP-UX
@@ -167,7 +170,7 @@
// GTEST_HAS_TYPED_TEST - typed tests
// GTEST_HAS_TYPED_TEST_P - type-parameterized tests
// GTEST_IS_THREADSAFE - Google Test is thread-safe.
-// GOOGLETEST_CM0007 DO NOT DELETE
+// GTEST_USES_RE2 - the RE2 regular expression library is used
// GTEST_USES_POSIX_RE - enhanced POSIX regex is used. Do not confuse with
// GTEST_HAS_POSIX_RE (see above) which users can
// define themselves.
@@ -190,10 +193,6 @@
// GTEST_AMBIGUOUS_ELSE_BLOCKER_ - for disabling a gcc warning.
// GTEST_ATTRIBUTE_UNUSED_ - declares that a class' instances or a
// variable don't have to be used.
-// GTEST_DISALLOW_ASSIGN_ - disables copy operator=.
-// GTEST_DISALLOW_COPY_AND_ASSIGN_ - disables copy ctor and operator=.
-// GTEST_DISALLOW_MOVE_ASSIGN_ - disables move operator=.
-// GTEST_DISALLOW_MOVE_AND_ASSIGN_ - disables move ctor and operator=.
// GTEST_MUST_USE_RESULT_ - declares that a function's result must be used.
// GTEST_INTENTIONAL_CONST_COND_PUSH_ - start code section where MSVC C4127 is
// suppressed (constant conditional).
@@ -217,11 +216,13 @@
// - synchronization primitives.
//
// Regular expressions:
-// RE - a simple regular expression class using the POSIX
-// Extended Regular Expression syntax on UNIX-like platforms
-// GOOGLETEST_CM0008 DO NOT DELETE
-// or a reduced regular exception syntax on other
-// platforms, including Windows.
+// RE - a simple regular expression class using
+// 1) the RE2 syntax on all platforms when built with RE2
+// and Abseil as dependencies
+// 2) the POSIX Extended Regular Expression syntax on
+// UNIX-like platforms,
+// 3) A reduced regular exception syntax on other platforms,
+// including Windows.
// Logging:
// GTEST_LOG_() - logs messages at the specified severity level.
// LogToStderr() - directs all log messages to stderr.
@@ -241,8 +242,6 @@
// BiggestInt - the biggest signed integer type.
//
// Command-line utilities:
-// GTEST_DECLARE_*() - declares a flag.
-// GTEST_DEFINE_*() - defines a flag.
// GetInjectableArgvs() - returns the command line as a vector of strings.
//
// Environment variable utilities:
@@ -263,48 +262,55 @@
#include <string.h>
#include <cerrno>
+// #include <condition_variable> // Guarded by GTEST_IS_THREADSAFE below
#include <cstdint>
+#include <iostream>
#include <limits>
+#include <locale>
+#include <memory>
+#include <string>
+// #include <mutex> // Guarded by GTEST_IS_THREADSAFE below
+#include <tuple>
#include <type_traits>
+#include <vector>
#ifndef _WIN32_WCE
-# include <sys/types.h>
-# include <sys/stat.h>
+#include <sys/stat.h>
+#include <sys/types.h>
#endif // !_WIN32_WCE
#if defined __APPLE__
-# include <AvailabilityMacros.h>
-# include <TargetConditionals.h>
+#include <AvailabilityMacros.h>
+#include <TargetConditionals.h>
#endif
-#include <iostream> // NOLINT
-#include <locale>
-#include <memory>
-#include <string> // NOLINT
-#include <tuple>
-#include <vector> // NOLINT
-
#include "gtest/internal/custom/gtest-port.h"
#include "gtest/internal/gtest-port-arch.h"
+#if GTEST_HAS_ABSL
+#include "absl/flags/declare.h"
+#include "absl/flags/flag.h"
+#include "absl/flags/reflection.h"
+#endif
+
#if !defined(GTEST_DEV_EMAIL_)
-# define GTEST_DEV_EMAIL_ "googletestframework@@googlegroups.com"
-# define GTEST_FLAG_PREFIX_ "gtest_"
-# define GTEST_FLAG_PREFIX_DASH_ "gtest-"
-# define GTEST_FLAG_PREFIX_UPPER_ "GTEST_"
-# define GTEST_NAME_ "Google Test"
-# define GTEST_PROJECT_URL_ "https://github.com/google/googletest/"
+#define GTEST_DEV_EMAIL_ "googletestframework@@googlegroups.com"
+#define GTEST_FLAG_PREFIX_ "gtest_"
+#define GTEST_FLAG_PREFIX_DASH_ "gtest-"
+#define GTEST_FLAG_PREFIX_UPPER_ "GTEST_"
+#define GTEST_NAME_ "Google Test"
+#define GTEST_PROJECT_URL_ "https://github.com/google/googletest/"
#endif // !defined(GTEST_DEV_EMAIL_)
#if !defined(GTEST_INIT_GOOGLE_TEST_NAME_)
-# define GTEST_INIT_GOOGLE_TEST_NAME_ "testing::InitGoogleTest"
+#define GTEST_INIT_GOOGLE_TEST_NAME_ "testing::InitGoogleTest"
#endif // !defined(GTEST_INIT_GOOGLE_TEST_NAME_)
// Determines the version of gcc that is used to compile this.
#ifdef __GNUC__
// 40302 means version 4.3.2.
-# define GTEST_GCC_VER_ \
- (__GNUC__*10000 + __GNUC_MINOR__*100 + __GNUC_PATCHLEVEL__)
+#define GTEST_GCC_VER_ \
+ (__GNUC__ * 10000 + __GNUC_MINOR__ * 100 + __GNUC_PATCHLEVEL__)
#endif // __GNUC__
// Macros for disabling Microsoft Visual C++ warnings.
@@ -313,41 +319,37 @@
// /* code that triggers warnings C4800 and C4385 */
// GTEST_DISABLE_MSC_WARNINGS_POP_()
#if defined(_MSC_VER)
-# define GTEST_DISABLE_MSC_WARNINGS_PUSH_(warnings) \
- __pragma(warning(push)) \
- __pragma(warning(disable: warnings))
-# define GTEST_DISABLE_MSC_WARNINGS_POP_() \
- __pragma(warning(pop))
+#define GTEST_DISABLE_MSC_WARNINGS_PUSH_(warnings) \
+ __pragma(warning(push)) __pragma(warning(disable : warnings))
+#define GTEST_DISABLE_MSC_WARNINGS_POP_() __pragma(warning(pop))
#else
// Not all compilers are MSVC
-# define GTEST_DISABLE_MSC_WARNINGS_PUSH_(warnings)
-# define GTEST_DISABLE_MSC_WARNINGS_POP_()
+#define GTEST_DISABLE_MSC_WARNINGS_PUSH_(warnings)
+#define GTEST_DISABLE_MSC_WARNINGS_POP_()
#endif
// Clang on Windows does not understand MSVC's pragma warning.
// We need clang-specific way to disable function deprecation warning.
#ifdef __clang__
-# define GTEST_DISABLE_MSC_DEPRECATED_PUSH_() \
- _Pragma("clang diagnostic push") \
- _Pragma("clang diagnostic ignored \"-Wdeprecated-declarations\"") \
- _Pragma("clang diagnostic ignored \"-Wdeprecated-implementations\"")
-#define GTEST_DISABLE_MSC_DEPRECATED_POP_() \
- _Pragma("clang diagnostic pop")
+#define GTEST_DISABLE_MSC_DEPRECATED_PUSH_() \
+ _Pragma("clang diagnostic push") \
+ _Pragma("clang diagnostic ignored \"-Wdeprecated-declarations\"") \
+ _Pragma("clang diagnostic ignored \"-Wdeprecated-implementations\"")
+#define GTEST_DISABLE_MSC_DEPRECATED_POP_() _Pragma("clang diagnostic pop")
#else
-# define GTEST_DISABLE_MSC_DEPRECATED_PUSH_() \
- GTEST_DISABLE_MSC_WARNINGS_PUSH_(4996)
-# define GTEST_DISABLE_MSC_DEPRECATED_POP_() \
- GTEST_DISABLE_MSC_WARNINGS_POP_()
+#define GTEST_DISABLE_MSC_DEPRECATED_PUSH_() \
+ GTEST_DISABLE_MSC_WARNINGS_PUSH_(4996)
+#define GTEST_DISABLE_MSC_DEPRECATED_POP_() GTEST_DISABLE_MSC_WARNINGS_POP_()
#endif
// Brings in definitions for functions used in the testing::internal::posix
// namespace (read, write, close, chdir, isatty, stat). We do not currently
// use them on Windows Mobile.
#if GTEST_OS_WINDOWS
-# if !GTEST_OS_WINDOWS_MOBILE
-# include <direct.h>
-# include <io.h>
-# endif
+#if !GTEST_OS_WINDOWS_MOBILE
+#include <direct.h>
+#include <io.h>
+#endif
// In order to avoid having to include <windows.h>, use forward declaration
#if GTEST_OS_WINDOWS_MINGW && !defined(__MINGW64_VERSION_MAJOR)
// MinGW defined _CRITICAL_SECTION and _RTL_CRITICAL_SECTION as two
@@ -367,68 +369,55 @@
// This assumes that non-Windows OSes provide unistd.h. For OSes where this
// is not the case, we need to include headers that provide the functions
// mentioned above.
-# include <unistd.h>
-# include <strings.h>
+#include <strings.h>
+#include <unistd.h>
#endif // GTEST_OS_WINDOWS
#if GTEST_OS_LINUX_ANDROID
// Used to define __ANDROID_API__ matching the target NDK API level.
-# include <android/api-level.h> // NOLINT
+#include <android/api-level.h> // NOLINT
#endif
// Defines this to true if and only if Google Test can use POSIX regular
// expressions.
#ifndef GTEST_HAS_POSIX_RE
-# if GTEST_OS_LINUX_ANDROID
+#if GTEST_OS_LINUX_ANDROID
// On Android, <regex.h> is only available starting with Gingerbread.
-# define GTEST_HAS_POSIX_RE (__ANDROID_API__ >= 9)
-# else
+#define GTEST_HAS_POSIX_RE (__ANDROID_API__ >= 9)
+#else
#define GTEST_HAS_POSIX_RE (!GTEST_OS_WINDOWS && !GTEST_OS_XTENSA)
-# endif
+#endif
#endif
-#if GTEST_USES_PCRE
-// The appropriate headers have already been included.
-
+// Select the regular expression implementation.
+#if GTEST_HAS_ABSL
+// When using Abseil, RE2 is required.
+#include "absl/strings/string_view.h"
+#include "re2/re2.h"
+#define GTEST_USES_RE2 1
#elif GTEST_HAS_POSIX_RE
-
-// On some platforms, <regex.h> needs someone to define size_t, and
-// won't compile otherwise. We can #include it here as we already
-// included <stdlib.h>, which is guaranteed to define size_t through
-// <stddef.h>.
-# include <regex.h> // NOLINT
-
-# define GTEST_USES_POSIX_RE 1
-
-#elif GTEST_OS_WINDOWS
-
-// <regex.h> is not available on Windows. Use our own simple regex
-// implementation instead.
-# define GTEST_USES_SIMPLE_RE 1
-
+#include <regex.h> // NOLINT
+#define GTEST_USES_POSIX_RE 1
#else
-
-// <regex.h> may not be available on this platform. Use our own
-// simple regex implementation instead.
-# define GTEST_USES_SIMPLE_RE 1
-
-#endif // GTEST_USES_PCRE
+// Use our own simple regex implementation.
+#define GTEST_USES_SIMPLE_RE 1
+#endif
#ifndef GTEST_HAS_EXCEPTIONS
// The user didn't tell us whether exceptions are enabled, so we need
// to figure it out.
-# if defined(_MSC_VER) && defined(_CPPUNWIND)
+#if defined(_MSC_VER) && defined(_CPPUNWIND)
// MSVC defines _CPPUNWIND to 1 if and only if exceptions are enabled.
-# define GTEST_HAS_EXCEPTIONS 1
-# elif defined(__BORLANDC__)
+#define GTEST_HAS_EXCEPTIONS 1
+#elif defined(__BORLANDC__)
// C++Builder's implementation of the STL uses the _HAS_EXCEPTIONS
// macro to enable exceptions, so we'll do the same.
// Assumes that exceptions are enabled by default.
-# ifndef _HAS_EXCEPTIONS
-# define _HAS_EXCEPTIONS 1
-# endif // _HAS_EXCEPTIONS
-# define GTEST_HAS_EXCEPTIONS _HAS_EXCEPTIONS
-# elif defined(__clang__)
+#ifndef _HAS_EXCEPTIONS
+#define _HAS_EXCEPTIONS 1
+#endif // _HAS_EXCEPTIONS
+#define GTEST_HAS_EXCEPTIONS _HAS_EXCEPTIONS
+#elif defined(__clang__)
// clang defines __EXCEPTIONS if and only if exceptions are enabled before clang
// 220714, but if and only if cleanups are enabled after that. In Obj-C++ files,
// there can be cleanups for ObjC exceptions which also need cleanups, even if
@@ -437,27 +426,27 @@
// cleanups prior to that. To reliably check for C++ exception availability with
// clang, check for
// __EXCEPTIONS && __has_feature(cxx_exceptions).
-# define GTEST_HAS_EXCEPTIONS (__EXCEPTIONS && __has_feature(cxx_exceptions))
-# elif defined(__GNUC__) && __EXCEPTIONS
+#define GTEST_HAS_EXCEPTIONS (__EXCEPTIONS && __has_feature(cxx_exceptions))
+#elif defined(__GNUC__) && __EXCEPTIONS
// gcc defines __EXCEPTIONS to 1 if and only if exceptions are enabled.
-# define GTEST_HAS_EXCEPTIONS 1
-# elif defined(__SUNPRO_CC)
+#define GTEST_HAS_EXCEPTIONS 1
+#elif defined(__SUNPRO_CC)
// Sun Pro CC supports exceptions. However, there is no compile-time way of
// detecting whether they are enabled or not. Therefore, we assume that
// they are enabled unless the user tells us otherwise.
-# define GTEST_HAS_EXCEPTIONS 1
-# elif defined(__IBMCPP__) && __EXCEPTIONS
+#define GTEST_HAS_EXCEPTIONS 1
+#elif defined(__IBMCPP__) && __EXCEPTIONS
// xlC defines __EXCEPTIONS to 1 if and only if exceptions are enabled.
-# define GTEST_HAS_EXCEPTIONS 1
-# elif defined(__HP_aCC)
+#define GTEST_HAS_EXCEPTIONS 1
+#elif defined(__HP_aCC)
// Exception handling is in effect by default in HP aCC compiler. It has to
// be turned of by +noeh compiler option if desired.
-# define GTEST_HAS_EXCEPTIONS 1
-# else
+#define GTEST_HAS_EXCEPTIONS 1
+#else
// For other compilers, we assume exceptions are disabled to be
// conservative.
-# define GTEST_HAS_EXCEPTIONS 0
-# endif // defined(_MSC_VER) || defined(__BORLANDC__)
+#define GTEST_HAS_EXCEPTIONS 0
+#endif // defined(_MSC_VER) || defined(__BORLANDC__)
#endif // GTEST_HAS_EXCEPTIONS
#ifndef GTEST_HAS_STD_WSTRING
@@ -477,63 +466,62 @@
// The user didn't tell us whether RTTI is enabled, so we need to
// figure it out.
-# ifdef _MSC_VER
+#ifdef _MSC_VER
#ifdef _CPPRTTI // MSVC defines this macro if and only if RTTI is enabled.
-# define GTEST_HAS_RTTI 1
-# else
-# define GTEST_HAS_RTTI 0
-# endif
+#define GTEST_HAS_RTTI 1
+#else
+#define GTEST_HAS_RTTI 0
+#endif
// Starting with version 4.3.2, gcc defines __GXX_RTTI if and only if RTTI is
// enabled.
-# elif defined(__GNUC__)
+#elif defined(__GNUC__)
-# ifdef __GXX_RTTI
+#ifdef __GXX_RTTI
// When building against STLport with the Android NDK and with
// -frtti -fno-exceptions, the build fails at link time with undefined
// references to __cxa_bad_typeid. Note sure if STL or toolchain bug,
// so disable RTTI when detected.
-# if GTEST_OS_LINUX_ANDROID && defined(_STLPORT_MAJOR) && \
- !defined(__EXCEPTIONS)
-# define GTEST_HAS_RTTI 0
-# else
-# define GTEST_HAS_RTTI 1
-# endif // GTEST_OS_LINUX_ANDROID && __STLPORT_MAJOR && !__EXCEPTIONS
-# else
-# define GTEST_HAS_RTTI 0
-# endif // __GXX_RTTI
+#if GTEST_OS_LINUX_ANDROID && defined(_STLPORT_MAJOR) && !defined(__EXCEPTIONS)
+#define GTEST_HAS_RTTI 0
+#else
+#define GTEST_HAS_RTTI 1
+#endif // GTEST_OS_LINUX_ANDROID && __STLPORT_MAJOR && !__EXCEPTIONS
+#else
+#define GTEST_HAS_RTTI 0
+#endif // __GXX_RTTI
// Clang defines __GXX_RTTI starting with version 3.0, but its manual recommends
// using has_feature instead. has_feature(cxx_rtti) is supported since 2.7, the
// first version with C++ support.
-# elif defined(__clang__)
+#elif defined(__clang__)
-# define GTEST_HAS_RTTI __has_feature(cxx_rtti)
+#define GTEST_HAS_RTTI __has_feature(cxx_rtti)
// Starting with version 9.0 IBM Visual Age defines __RTTI_ALL__ to 1 if
// both the typeid and dynamic_cast features are present.
-# elif defined(__IBMCPP__) && (__IBMCPP__ >= 900)
+#elif defined(__IBMCPP__) && (__IBMCPP__ >= 900)
-# ifdef __RTTI_ALL__
-# define GTEST_HAS_RTTI 1
-# else
-# define GTEST_HAS_RTTI 0
-# endif
+#ifdef __RTTI_ALL__
+#define GTEST_HAS_RTTI 1
+#else
+#define GTEST_HAS_RTTI 0
+#endif
-# else
+#else
// For all other compilers, we assume RTTI is enabled.
-# define GTEST_HAS_RTTI 1
+#define GTEST_HAS_RTTI 1
-# endif // _MSC_VER
+#endif // _MSC_VER
#endif // GTEST_HAS_RTTI
// It's this header's responsibility to #include <typeinfo> when RTTI
// is enabled.
#if GTEST_HAS_RTTI
-# include <typeinfo>
+#include <typeinfo>
#endif
// Determines whether Google Test can use the pthreads library.
@@ -547,16 +535,16 @@
(GTEST_OS_LINUX || GTEST_OS_MAC || GTEST_OS_HPUX || GTEST_OS_QNX || \
GTEST_OS_FREEBSD || GTEST_OS_NACL || GTEST_OS_NETBSD || GTEST_OS_FUCHSIA || \
GTEST_OS_DRAGONFLY || GTEST_OS_GNU_KFREEBSD || GTEST_OS_OPENBSD || \
- GTEST_OS_HAIKU)
+ GTEST_OS_HAIKU || GTEST_OS_GNU_HURD)
#endif // GTEST_HAS_PTHREAD
#if GTEST_HAS_PTHREAD
// gtest-port.h guarantees to #include <pthread.h> when GTEST_HAS_PTHREAD is
// true.
-# include <pthread.h> // NOLINT
+#include <pthread.h> // NOLINT
// For timespec and nanosleep, used below.
-# include <time.h> // NOLINT
+#include <time.h> // NOLINT
#endif
// Determines whether clone(2) is supported.
@@ -566,24 +554,23 @@
#ifndef GTEST_HAS_CLONE
// The user didn't tell us, so we need to figure it out.
-# if GTEST_OS_LINUX && !defined(__ia64__)
-# if GTEST_OS_LINUX_ANDROID
+#if GTEST_OS_LINUX && !defined(__ia64__)
+#if GTEST_OS_LINUX_ANDROID
// On Android, clone() became available at different API levels for each 32-bit
// architecture.
-# if defined(__LP64__) || \
- (defined(__arm__) && __ANDROID_API__ >= 9) || \
- (defined(__mips__) && __ANDROID_API__ >= 12) || \
- (defined(__i386__) && __ANDROID_API__ >= 17)
-# define GTEST_HAS_CLONE 1
-# else
-# define GTEST_HAS_CLONE 0
-# endif
-# else
-# define GTEST_HAS_CLONE 1
-# endif
-# else
-# define GTEST_HAS_CLONE 0
-# endif // GTEST_OS_LINUX && !defined(__ia64__)
+#if defined(__LP64__) || (defined(__arm__) && __ANDROID_API__ >= 9) || \
+ (defined(__mips__) && __ANDROID_API__ >= 12) || \
+ (defined(__i386__) && __ANDROID_API__ >= 17)
+#define GTEST_HAS_CLONE 1
+#else
+#define GTEST_HAS_CLONE 0
+#endif
+#else
+#define GTEST_HAS_CLONE 1
+#endif
+#else
+#define GTEST_HAS_CLONE 0
+#endif // GTEST_OS_LINUX && !defined(__ia64__)
#endif // GTEST_HAS_CLONE
@@ -594,10 +581,10 @@
// platforms except known mobile ones.
#if GTEST_OS_WINDOWS_MOBILE || GTEST_OS_WINDOWS_PHONE || \
GTEST_OS_WINDOWS_RT || GTEST_OS_ESP8266 || GTEST_OS_XTENSA
-# define GTEST_HAS_STREAM_REDIRECTION 0
-# else
-# define GTEST_HAS_STREAM_REDIRECTION 1
-# endif // !GTEST_OS_WINDOWS_MOBILE
+#define GTEST_HAS_STREAM_REDIRECTION 0
+#else
+#define GTEST_HAS_STREAM_REDIRECTION 1
+#endif // !GTEST_OS_WINDOWS_MOBILE
#endif // GTEST_HAS_STREAM_REDIRECTION
// Determines whether to support death tests.
@@ -607,8 +594,9 @@
(GTEST_OS_WINDOWS_DESKTOP && _MSC_VER) || GTEST_OS_WINDOWS_MINGW || \
GTEST_OS_AIX || GTEST_OS_HPUX || GTEST_OS_OPENBSD || GTEST_OS_QNX || \
GTEST_OS_FREEBSD || GTEST_OS_NETBSD || GTEST_OS_FUCHSIA || \
- GTEST_OS_DRAGONFLY || GTEST_OS_GNU_KFREEBSD || GTEST_OS_HAIKU)
-# define GTEST_HAS_DEATH_TEST 1
+ GTEST_OS_DRAGONFLY || GTEST_OS_GNU_KFREEBSD || GTEST_OS_HAIKU || \
+ GTEST_OS_GNU_HURD)
+#define GTEST_HAS_DEATH_TEST 1
#endif
// Determines whether to support type-driven tests.
@@ -617,8 +605,8 @@
// Sun Pro CC, IBM Visual Age, and HP aCC support.
#if defined(__GNUC__) || defined(_MSC_VER) || defined(__SUNPRO_CC) || \
defined(__IBMCPP__) || defined(__HP_aCC)
-# define GTEST_HAS_TYPED_TEST 1
-# define GTEST_HAS_TYPED_TEST_P 1
+#define GTEST_HAS_TYPED_TEST 1
+#define GTEST_HAS_TYPED_TEST_P 1
#endif
// Determines whether the system compiler uses UTF-16 for encoding wide strings.
@@ -627,8 +615,9 @@
// Determines whether test results can be streamed to a socket.
#if GTEST_OS_LINUX || GTEST_OS_GNU_KFREEBSD || GTEST_OS_DRAGONFLY || \
- GTEST_OS_FREEBSD || GTEST_OS_NETBSD || GTEST_OS_OPENBSD
-# define GTEST_CAN_STREAM_RESULTS_ 1
+ GTEST_OS_FREEBSD || GTEST_OS_NETBSD || GTEST_OS_OPENBSD || \
+ GTEST_OS_GNU_HURD
+#define GTEST_CAN_STREAM_RESULTS_ 1
#endif
// Defines some utility macros.
@@ -642,9 +631,12 @@
//
// The "switch (0) case 0:" idiom is used to suppress this.
#ifdef __INTEL_COMPILER
-# define GTEST_AMBIGUOUS_ELSE_BLOCKER_
+#define GTEST_AMBIGUOUS_ELSE_BLOCKER_
#else
-# define GTEST_AMBIGUOUS_ELSE_BLOCKER_ switch (0) case 0: default: // NOLINT
+#define GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
+ switch (0) \
+ case 0: \
+ default: // NOLINT
#endif
// Use this annotation at the end of a struct/class definition to
@@ -659,55 +651,32 @@
// Also use it after a variable or parameter declaration to tell the
// compiler the variable/parameter does not have to be used.
#if defined(__GNUC__) && !defined(COMPILER_ICC)
-# define GTEST_ATTRIBUTE_UNUSED_ __attribute__ ((unused))
+#define GTEST_ATTRIBUTE_UNUSED_ __attribute__((unused))
#elif defined(__clang__)
-# if __has_attribute(unused)
-# define GTEST_ATTRIBUTE_UNUSED_ __attribute__ ((unused))
-# endif
+#if __has_attribute(unused)
+#define GTEST_ATTRIBUTE_UNUSED_ __attribute__((unused))
+#endif
#endif
#ifndef GTEST_ATTRIBUTE_UNUSED_
-# define GTEST_ATTRIBUTE_UNUSED_
+#define GTEST_ATTRIBUTE_UNUSED_
#endif
// Use this annotation before a function that takes a printf format string.
#if (defined(__GNUC__) || defined(__clang__)) && !defined(COMPILER_ICC)
-# if defined(__MINGW_PRINTF_FORMAT)
+#if defined(__MINGW_PRINTF_FORMAT)
// MinGW has two different printf implementations. Ensure the format macro
// matches the selected implementation. See
// https://sourceforge.net/p/mingw-w64/wiki2/gnu%20printf/.
-# define GTEST_ATTRIBUTE_PRINTF_(string_index, first_to_check) \
- __attribute__((__format__(__MINGW_PRINTF_FORMAT, string_index, \
- first_to_check)))
-# else
-# define GTEST_ATTRIBUTE_PRINTF_(string_index, first_to_check) \
- __attribute__((__format__(__printf__, string_index, first_to_check)))
-# endif
+#define GTEST_ATTRIBUTE_PRINTF_(string_index, first_to_check) \
+ __attribute__(( \
+ __format__(__MINGW_PRINTF_FORMAT, string_index, first_to_check)))
#else
-# define GTEST_ATTRIBUTE_PRINTF_(string_index, first_to_check)
+#define GTEST_ATTRIBUTE_PRINTF_(string_index, first_to_check) \
+ __attribute__((__format__(__printf__, string_index, first_to_check)))
#endif
-
-
-// A macro to disallow copy operator=
-// This should be used in the private: declarations for a class.
-#define GTEST_DISALLOW_ASSIGN_(type) \
- type& operator=(type const &) = delete
-
-// A macro to disallow copy constructor and operator=
-// This should be used in the private: declarations for a class.
-#define GTEST_DISALLOW_COPY_AND_ASSIGN_(type) \
- type(type const&) = delete; \
- type& operator=(type const&) = delete
-
-// A macro to disallow move operator=
-// This should be used in the private: declarations for a class.
-#define GTEST_DISALLOW_MOVE_ASSIGN_(type) \
- type& operator=(type &&) noexcept = delete
-
-// A macro to disallow move constructor and operator=
-// This should be used in the private: declarations for a class.
-#define GTEST_DISALLOW_MOVE_AND_ASSIGN_(type) \
- type(type&&) noexcept = delete; \
- type& operator=(type&&) noexcept = delete
+#else
+#define GTEST_ATTRIBUTE_PRINTF_(string_index, first_to_check)
+#endif
// Tell the compiler to warn about unused return values for functions declared
// with this macro. The macro should be used on function declarations
@@ -715,9 +684,9 @@
//
// Sprocket* AllocateSprocket() GTEST_MUST_USE_RESULT_;
#if defined(__GNUC__) && !defined(COMPILER_ICC)
-# define GTEST_MUST_USE_RESULT_ __attribute__ ((warn_unused_result))
+#define GTEST_MUST_USE_RESULT_ __attribute__((warn_unused_result))
#else
-# define GTEST_MUST_USE_RESULT_
+#define GTEST_MUST_USE_RESULT_
#endif // __GNUC__ && !COMPILER_ICC
// MS C++ compiler emits warning when a conditional expression is compile time
@@ -728,10 +697,9 @@
// while (true) {
// GTEST_INTENTIONAL_CONST_COND_POP_()
// }
-# define GTEST_INTENTIONAL_CONST_COND_PUSH_() \
- GTEST_DISABLE_MSC_WARNINGS_PUSH_(4127)
-# define GTEST_INTENTIONAL_CONST_COND_POP_() \
- GTEST_DISABLE_MSC_WARNINGS_POP_()
+#define GTEST_INTENTIONAL_CONST_COND_PUSH_() \
+ GTEST_DISABLE_MSC_WARNINGS_PUSH_(4127)
+#define GTEST_INTENTIONAL_CONST_COND_POP_() GTEST_DISABLE_MSC_WARNINGS_POP_()
// Determine whether the compiler supports Microsoft's Structured Exception
// Handling. This is supported by several Windows compilers but generally
@@ -739,13 +707,13 @@
#ifndef GTEST_HAS_SEH
// The user didn't tell us, so we need to figure it out.
-# if defined(_MSC_VER) || defined(__BORLANDC__)
+#if defined(_MSC_VER) || defined(__BORLANDC__)
// These two compilers are known to support SEH.
-# define GTEST_HAS_SEH 1
-# else
+#define GTEST_HAS_SEH 1
+#else
// Assume no SEH.
-# define GTEST_HAS_SEH 0
-# endif
+#define GTEST_HAS_SEH 0
+#endif
#endif // GTEST_HAS_SEH
@@ -758,94 +726,112 @@
#endif // GTEST_IS_THREADSAFE
+#if GTEST_IS_THREADSAFE
+// Some platforms don't support including these threading related headers.
+#include <condition_variable> // NOLINT
+#include <mutex> // NOLINT
+#endif // GTEST_IS_THREADSAFE
+
// GTEST_API_ qualifies all symbols that must be exported. The definitions below
// are guarded by #ifndef to give embedders a chance to define GTEST_API_ in
// gtest/internal/custom/gtest-port.h
#ifndef GTEST_API_
#ifdef _MSC_VER
-# if GTEST_LINKED_AS_SHARED_LIBRARY
-# define GTEST_API_ __declspec(dllimport)
-# elif GTEST_CREATE_SHARED_LIBRARY
-# define GTEST_API_ __declspec(dllexport)
-# endif
+#if GTEST_LINKED_AS_SHARED_LIBRARY
+#define GTEST_API_ __declspec(dllimport)
+#elif GTEST_CREATE_SHARED_LIBRARY
+#define GTEST_API_ __declspec(dllexport)
+#endif
#elif __GNUC__ >= 4 || defined(__clang__)
-# define GTEST_API_ __attribute__((visibility ("default")))
+#define GTEST_API_ __attribute__((visibility("default")))
#endif // _MSC_VER
#endif // GTEST_API_
#ifndef GTEST_API_
-# define GTEST_API_
+#define GTEST_API_
#endif // GTEST_API_
#ifndef GTEST_DEFAULT_DEATH_TEST_STYLE
-# define GTEST_DEFAULT_DEATH_TEST_STYLE "fast"
+#define GTEST_DEFAULT_DEATH_TEST_STYLE "fast"
#endif // GTEST_DEFAULT_DEATH_TEST_STYLE
#ifdef __GNUC__
// Ask the compiler to never inline a given function.
-# define GTEST_NO_INLINE_ __attribute__((noinline))
+#define GTEST_NO_INLINE_ __attribute__((noinline))
#else
-# define GTEST_NO_INLINE_
+#define GTEST_NO_INLINE_
+#endif
+
+#if defined(__clang__)
+// Nested ifs to avoid triggering MSVC warning.
+#if __has_attribute(disable_tail_calls)
+// Ask the compiler not to perform tail call optimization inside
+// the marked function.
+#define GTEST_NO_TAIL_CALL_ __attribute__((disable_tail_calls))
+#endif
+#elif __GNUC__
+#define GTEST_NO_TAIL_CALL_ \
+ __attribute__((optimize("no-optimize-sibling-calls")))
+#else
+#define GTEST_NO_TAIL_CALL_
#endif
// _LIBCPP_VERSION is defined by the libc++ library from the LLVM project.
#if !defined(GTEST_HAS_CXXABI_H_)
-# if defined(__GLIBCXX__) || (defined(_LIBCPP_VERSION) && !defined(_MSC_VER))
-# define GTEST_HAS_CXXABI_H_ 1
-# else
-# define GTEST_HAS_CXXABI_H_ 0
-# endif
+#if defined(__GLIBCXX__) || (defined(_LIBCPP_VERSION) && !defined(_MSC_VER))
+#define GTEST_HAS_CXXABI_H_ 1
+#else
+#define GTEST_HAS_CXXABI_H_ 0
+#endif
#endif
// A function level attribute to disable checking for use of uninitialized
// memory when built with MemorySanitizer.
#if defined(__clang__)
-# if __has_feature(memory_sanitizer)
-# define GTEST_ATTRIBUTE_NO_SANITIZE_MEMORY_ \
- __attribute__((no_sanitize_memory))
-# else
-# define GTEST_ATTRIBUTE_NO_SANITIZE_MEMORY_
-# endif // __has_feature(memory_sanitizer)
+#if __has_feature(memory_sanitizer)
+#define GTEST_ATTRIBUTE_NO_SANITIZE_MEMORY_ __attribute__((no_sanitize_memory))
#else
-# define GTEST_ATTRIBUTE_NO_SANITIZE_MEMORY_
+#define GTEST_ATTRIBUTE_NO_SANITIZE_MEMORY_
+#endif // __has_feature(memory_sanitizer)
+#else
+#define GTEST_ATTRIBUTE_NO_SANITIZE_MEMORY_
#endif // __clang__
// A function level attribute to disable AddressSanitizer instrumentation.
#if defined(__clang__)
-# if __has_feature(address_sanitizer)
-# define GTEST_ATTRIBUTE_NO_SANITIZE_ADDRESS_ \
- __attribute__((no_sanitize_address))
-# else
-# define GTEST_ATTRIBUTE_NO_SANITIZE_ADDRESS_
-# endif // __has_feature(address_sanitizer)
+#if __has_feature(address_sanitizer)
+#define GTEST_ATTRIBUTE_NO_SANITIZE_ADDRESS_ \
+ __attribute__((no_sanitize_address))
#else
-# define GTEST_ATTRIBUTE_NO_SANITIZE_ADDRESS_
+#define GTEST_ATTRIBUTE_NO_SANITIZE_ADDRESS_
+#endif // __has_feature(address_sanitizer)
+#else
+#define GTEST_ATTRIBUTE_NO_SANITIZE_ADDRESS_
#endif // __clang__
// A function level attribute to disable HWAddressSanitizer instrumentation.
#if defined(__clang__)
-# if __has_feature(hwaddress_sanitizer)
-# define GTEST_ATTRIBUTE_NO_SANITIZE_HWADDRESS_ \
- __attribute__((no_sanitize("hwaddress")))
-# else
-# define GTEST_ATTRIBUTE_NO_SANITIZE_HWADDRESS_
-# endif // __has_feature(hwaddress_sanitizer)
+#if __has_feature(hwaddress_sanitizer)
+#define GTEST_ATTRIBUTE_NO_SANITIZE_HWADDRESS_ \
+ __attribute__((no_sanitize("hwaddress")))
#else
-# define GTEST_ATTRIBUTE_NO_SANITIZE_HWADDRESS_
+#define GTEST_ATTRIBUTE_NO_SANITIZE_HWADDRESS_
+#endif // __has_feature(hwaddress_sanitizer)
+#else
+#define GTEST_ATTRIBUTE_NO_SANITIZE_HWADDRESS_
#endif // __clang__
// A function level attribute to disable ThreadSanitizer instrumentation.
#if defined(__clang__)
-# if __has_feature(thread_sanitizer)
-# define GTEST_ATTRIBUTE_NO_SANITIZE_THREAD_ \
- __attribute__((no_sanitize_thread))
-# else
-# define GTEST_ATTRIBUTE_NO_SANITIZE_THREAD_
-# endif // __has_feature(thread_sanitizer)
+#if __has_feature(thread_sanitizer)
+#define GTEST_ATTRIBUTE_NO_SANITIZE_THREAD_ __attribute__((no_sanitize_thread))
#else
-# define GTEST_ATTRIBUTE_NO_SANITIZE_THREAD_
+#define GTEST_ATTRIBUTE_NO_SANITIZE_THREAD_
+#endif // __has_feature(thread_sanitizer)
+#else
+#define GTEST_ATTRIBUTE_NO_SANITIZE_THREAD_
#endif // __clang__
namespace testing {
@@ -867,25 +853,37 @@
// Secret object, which is what we want.
class Secret;
-// The GTEST_COMPILE_ASSERT_ is a legacy macro used to verify that a compile
-// time expression is true (in new code, use static_assert instead). For
-// example, you could use it to verify the size of a static array:
-//
-// GTEST_COMPILE_ASSERT_(GTEST_ARRAY_SIZE_(names) == NUM_NAMES,
-// names_incorrect_size);
-//
-// The second argument to the macro must be a valid C++ identifier. If the
-// expression is false, compiler will issue an error containing this identifier.
-#define GTEST_COMPILE_ASSERT_(expr, msg) static_assert(expr, #msg)
-
// A helper for suppressing warnings on constant condition. It just
// returns 'condition'.
GTEST_API_ bool IsTrue(bool condition);
// Defines RE.
-#if GTEST_USES_PCRE
-// if used, PCRE is injected by custom/gtest-port.h
+#if GTEST_USES_RE2
+
+// This is almost `using RE = ::RE2`, except it is copy-constructible, and it
+// needs to disambiguate the `std::string`, `absl::string_view`, and `const
+// char*` constructors.
+class GTEST_API_ RE {
+ public:
+ RE(absl::string_view regex) : regex_(regex) {} // NOLINT
+ RE(const char* regex) : RE(absl::string_view(regex)) {} // NOLINT
+ RE(const std::string& regex) : RE(absl::string_view(regex)) {} // NOLINT
+ RE(const RE& other) : RE(other.pattern()) {}
+
+ const std::string& pattern() const { return regex_.pattern(); }
+
+ static bool FullMatch(absl::string_view str, const RE& re) {
+ return RE2::FullMatch(str, re.regex_);
+ }
+ static bool PartialMatch(absl::string_view str, const RE& re) {
+ return RE2::PartialMatch(str, re.regex_);
+ }
+
+ private:
+ RE2 regex_;
+};
+
#elif GTEST_USES_POSIX_RE || GTEST_USES_SIMPLE_RE
// A simple C++ wrapper for <regex.h>. It uses the POSIX Extended
@@ -924,19 +922,19 @@
const char* pattern_;
bool is_valid_;
-# if GTEST_USES_POSIX_RE
+#if GTEST_USES_POSIX_RE
regex_t full_regex_; // For FullMatch().
regex_t partial_regex_; // For PartialMatch().
-# else // GTEST_USES_SIMPLE_RE
+#else // GTEST_USES_SIMPLE_RE
const char* full_pattern_; // For FullMatch();
-# endif
+#endif
};
-#endif // GTEST_USES_PCRE
+#endif // ::testing::internal::RE implementation
// Formats a source file path and a line number as they would appear
// in an error message from the compiler used to compile this code.
@@ -954,12 +952,7 @@
// LogToStderr() - directs all log messages to stderr.
// FlushInfoLog() - flushes informational log messages.
-enum GTestLogSeverity {
- GTEST_INFO,
- GTEST_WARNING,
- GTEST_ERROR,
- GTEST_FATAL
-};
+enum GTestLogSeverity { GTEST_INFO, GTEST_WARNING, GTEST_ERROR, GTEST_FATAL };
// Formats log entry severity, provides a stream object for streaming the
// log message, and terminates the message with a newline when going out of
@@ -976,14 +969,16 @@
private:
const GTestLogSeverity severity_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(GTestLog);
+ GTestLog(const GTestLog&) = delete;
+ GTestLog& operator=(const GTestLog&) = delete;
};
#if !defined(GTEST_LOG_)
-# define GTEST_LOG_(severity) \
- ::testing::internal::GTestLog(::testing::internal::GTEST_##severity, \
- __FILE__, __LINE__).GetStream()
+#define GTEST_LOG_(severity) \
+ ::testing::internal::GTestLog(::testing::internal::GTEST_##severity, \
+ __FILE__, __LINE__) \
+ .GetStream()
inline void LogToStderr() {}
inline void FlushInfoLog() { fflush(nullptr); }
@@ -995,7 +990,7 @@
//
// GTEST_CHECK_ is an all-mode assert. It aborts the program if the condition
// is not satisfied.
-// Synopsys:
+// Synopsis:
// GTEST_CHECK_(boolean_condition);
// or
// GTEST_CHECK_(boolean_condition) << "Additional message";
@@ -1005,12 +1000,12 @@
// condition itself, plus additional message streamed into it, if any,
// and then it aborts the program. It aborts the program irrespective of
// whether it is built in the debug mode or not.
-# define GTEST_CHECK_(condition) \
- GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
- if (::testing::internal::IsTrue(condition)) \
- ; \
- else \
- GTEST_LOG_(FATAL) << "Condition " #condition " failed. "
+#define GTEST_CHECK_(condition) \
+ GTEST_AMBIGUOUS_ELSE_BLOCKER_ \
+ if (::testing::internal::IsTrue(condition)) \
+ ; \
+ else \
+ GTEST_LOG_(FATAL) << "Condition " #condition " failed. "
#endif // !defined(GTEST_CHECK_)
// An all-mode assert to verify that the given POSIX-style function
@@ -1019,9 +1014,8 @@
// in {} if you need to use it as the only statement in an 'if'
// branch.
#define GTEST_CHECK_POSIX_SUCCESS_(posix_call) \
- if (const int gtest_error = (posix_call)) \
- GTEST_LOG_(FATAL) << #posix_call << "failed with error " \
- << gtest_error
+ if (const int gtest_error = (posix_call)) \
+ GTEST_LOG_(FATAL) << #posix_call << "failed with error " << gtest_error
// Transforms "T" into "const T&" according to standard reference collapsing
// rules (this is only needed as a backport for C++98 compilers that do not
@@ -1035,9 +1029,13 @@
// Note that the non-const reference will not have "const" added. This is
// standard, and necessary so that "T" can always bind to "const T&".
template <typename T>
-struct ConstRef { typedef const T& type; };
+struct ConstRef {
+ typedef const T& type;
+};
template <typename T>
-struct ConstRef<T&> { typedef T& type; };
+struct ConstRef<T&> {
+ typedef T& type;
+};
// The argument T must depend on some template parameters.
#define GTEST_REFERENCE_TO_CONST_(T) \
@@ -1050,7 +1048,7 @@
// const Foo*). When you use ImplicitCast_, the compiler checks that
// the cast is safe. Such explicit ImplicitCast_s are necessary in
// surprisingly many situations where C++ demands an exact type match
-// instead of an argument type convertable to a target type.
+// instead of an argument type convertible to a target type.
//
// The syntax for using ImplicitCast_ is the same as for static_cast:
//
@@ -1063,8 +1061,10 @@
// This relatively ugly name is intentional. It prevents clashes with
// similar functions users may have (e.g., implicit_cast). The internal
// namespace alone is not enough because the function can be found by ADL.
-template<typename To>
-inline To ImplicitCast_(To x) { return x; }
+template <typename To>
+inline To ImplicitCast_(To x) {
+ return x;
+}
// When you upcast (that is, cast a pointer from type Foo to type
// SuperclassOfFoo), it's fine to use ImplicitCast_<>, since upcasts
@@ -1087,17 +1087,17 @@
// This relatively ugly name is intentional. It prevents clashes with
// similar functions users may have (e.g., down_cast). The internal
// namespace alone is not enough because the function can be found by ADL.
-template<typename To, typename From> // use like this: DownCast_<T*>(foo);
-inline To DownCast_(From* f) { // so we only accept pointers
+template <typename To, typename From> // use like this: DownCast_<T*>(foo);
+inline To DownCast_(From* f) { // so we only accept pointers
// Ensures that To is a sub-type of From *. This test is here only
// for compile-time type checking, and has no overhead in an
// optimized build at run-time, as it will be optimized away
// completely.
GTEST_INTENTIONAL_CONST_COND_PUSH_()
if (false) {
- GTEST_INTENTIONAL_CONST_COND_POP_()
- const To to = nullptr;
- ::testing::internal::ImplicitCast_<From*>(to);
+ GTEST_INTENTIONAL_CONST_COND_POP_()
+ const To to = nullptr;
+ ::testing::internal::ImplicitCast_<From*>(to);
}
#if GTEST_HAS_RTTI
@@ -1162,71 +1162,8 @@
// Defines synchronization primitives.
#if GTEST_IS_THREADSAFE
-# if GTEST_HAS_PTHREAD
-// Sleeps for (roughly) n milliseconds. This function is only for testing
-// Google Test's own constructs. Don't use it in user tests, either
-// directly or indirectly.
-inline void SleepMilliseconds(int n) {
- const timespec time = {
- 0, // 0 seconds.
- n * 1000L * 1000L, // And n ms.
- };
- nanosleep(&time, nullptr);
-}
-# endif // GTEST_HAS_PTHREAD
-# if GTEST_HAS_NOTIFICATION_
-// Notification has already been imported into the namespace.
-// Nothing to do here.
-
-# elif GTEST_HAS_PTHREAD
-// Allows a controller thread to pause execution of newly created
-// threads until notified. Instances of this class must be created
-// and destroyed in the controller thread.
-//
-// This class is only for testing Google Test's own constructs. Do not
-// use it in user tests, either directly or indirectly.
-class Notification {
- public:
- Notification() : notified_(false) {
- GTEST_CHECK_POSIX_SUCCESS_(pthread_mutex_init(&mutex_, nullptr));
- }
- ~Notification() {
- pthread_mutex_destroy(&mutex_);
- }
-
- // Notifies all threads created with this notification to start. Must
- // be called from the controller thread.
- void Notify() {
- pthread_mutex_lock(&mutex_);
- notified_ = true;
- pthread_mutex_unlock(&mutex_);
- }
-
- // Blocks until the controller thread notifies. Must be called from a test
- // thread.
- void WaitForNotification() {
- for (;;) {
- pthread_mutex_lock(&mutex_);
- const bool notified = notified_;
- pthread_mutex_unlock(&mutex_);
- if (notified)
- break;
- SleepMilliseconds(10);
- }
- }
-
- private:
- pthread_mutex_t mutex_;
- bool notified_;
-
- GTEST_DISALLOW_COPY_AND_ASSIGN_(Notification);
-};
-
-# elif GTEST_OS_WINDOWS && !GTEST_OS_WINDOWS_PHONE && !GTEST_OS_WINDOWS_RT
-
-GTEST_API_ void SleepMilliseconds(int n);
-
+#if GTEST_OS_WINDOWS
// Provides leak-safe Windows kernel handle ownership.
// Used in death tests and in threading support.
class GTEST_API_ AutoHandle {
@@ -1253,8 +1190,18 @@
Handle handle_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(AutoHandle);
+ AutoHandle(const AutoHandle&) = delete;
+ AutoHandle& operator=(const AutoHandle&) = delete;
};
+#endif
+
+#if GTEST_HAS_NOTIFICATION_
+// Notification has already been imported into the namespace.
+// Nothing to do here.
+
+#else
+GTEST_DISABLE_MSC_WARNINGS_PUSH_(4251 \
+/* class A needs to have dll-interface to be used by clients of class B */)
// Allows a controller thread to pause execution of newly created
// threads until notified. Instances of this class must be created
@@ -1262,23 +1209,40 @@
//
// This class is only for testing Google Test's own constructs. Do not
// use it in user tests, either directly or indirectly.
+// TODO(b/203539622): Replace unconditionally with absl::Notification.
class GTEST_API_ Notification {
public:
- Notification();
- void Notify();
- void WaitForNotification();
+ Notification() : notified_(false) {}
+ Notification(const Notification&) = delete;
+ Notification& operator=(const Notification&) = delete;
+
+ // Notifies all threads created with this notification to start. Must
+ // be called from the controller thread.
+ void Notify() {
+ std::lock_guard<std::mutex> lock(mu_);
+ notified_ = true;
+ cv_.notify_all();
+ }
+
+ // Blocks until the controller thread notifies. Must be called from a test
+ // thread.
+ void WaitForNotification() {
+ std::unique_lock<std::mutex> lock(mu_);
+ cv_.wait(lock, [this]() { return notified_; });
+ }
private:
- AutoHandle event_;
-
- GTEST_DISALLOW_COPY_AND_ASSIGN_(Notification);
+ std::mutex mu_;
+ std::condition_variable cv_;
+ bool notified_;
};
-# endif // GTEST_HAS_NOTIFICATION_
+GTEST_DISABLE_MSC_WARNINGS_POP_() // 4251
+#endif // GTEST_HAS_NOTIFICATION_
// On MinGW, we can have both GTEST_OS_WINDOWS and GTEST_HAS_PTHREAD
// defined, but we don't want to use MinGW's pthreads implementation, which
// has conformance problems with some versions of the POSIX standard.
-# if GTEST_HAS_PTHREAD && !GTEST_OS_WINDOWS_MINGW
+#if GTEST_HAS_PTHREAD && !GTEST_OS_WINDOWS_MINGW
// As a C-function, ThreadFuncWithCLinkage cannot be templated itself.
// Consequently, it cannot select a correct instantiation of ThreadWithParam
@@ -1354,16 +1318,17 @@
// finished.
pthread_t thread_; // The native thread object.
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ThreadWithParam);
+ ThreadWithParam(const ThreadWithParam&) = delete;
+ ThreadWithParam& operator=(const ThreadWithParam&) = delete;
};
-# endif // !GTEST_OS_WINDOWS && GTEST_HAS_PTHREAD ||
- // GTEST_HAS_MUTEX_AND_THREAD_LOCAL_
+#endif // !GTEST_OS_WINDOWS && GTEST_HAS_PTHREAD ||
+ // GTEST_HAS_MUTEX_AND_THREAD_LOCAL_
-# if GTEST_HAS_MUTEX_AND_THREAD_LOCAL_
+#if GTEST_HAS_MUTEX_AND_THREAD_LOCAL_
// Mutex and ThreadLocal have already been imported into the namespace.
// Nothing to do here.
-# elif GTEST_OS_WINDOWS && !GTEST_OS_WINDOWS_PHONE && !GTEST_OS_WINDOWS_RT
+#elif GTEST_OS_WINDOWS && !GTEST_OS_WINDOWS_PHONE && !GTEST_OS_WINDOWS_RT
// Mutex implements mutex on Windows platforms. It is used in conjunction
// with class MutexLock:
@@ -1417,14 +1382,15 @@
long critical_section_init_phase_; // NOLINT
GTEST_CRITICAL_SECTION* critical_section_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(Mutex);
+ Mutex(const Mutex&) = delete;
+ Mutex& operator=(const Mutex&) = delete;
};
-# define GTEST_DECLARE_STATIC_MUTEX_(mutex) \
- extern ::testing::internal::Mutex mutex
+#define GTEST_DECLARE_STATIC_MUTEX_(mutex) \
+ extern ::testing::internal::Mutex mutex
-# define GTEST_DEFINE_STATIC_MUTEX_(mutex) \
- ::testing::internal::Mutex mutex(::testing::internal::Mutex::kStaticMutex)
+#define GTEST_DEFINE_STATIC_MUTEX_(mutex) \
+ ::testing::internal::Mutex mutex(::testing::internal::Mutex::kStaticMutex)
// We cannot name this class MutexLock because the ctor declaration would
// conflict with a macro named MutexLock, which is defined on some
@@ -1433,15 +1399,15 @@
// "MutexLock l(&mu)". Hence the typedef trick below.
class GTestMutexLock {
public:
- explicit GTestMutexLock(Mutex* mutex)
- : mutex_(mutex) { mutex_->Lock(); }
+ explicit GTestMutexLock(Mutex* mutex) : mutex_(mutex) { mutex_->Lock(); }
~GTestMutexLock() { mutex_->Unlock(); }
private:
Mutex* const mutex_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(GTestMutexLock);
+ GTestMutexLock(const GTestMutexLock&) = delete;
+ GTestMutexLock& operator=(const GTestMutexLock&) = delete;
};
typedef GTestMutexLock MutexLock;
@@ -1468,7 +1434,8 @@
virtual ~ThreadLocalBase() {}
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ThreadLocalBase);
+ ThreadLocalBase(const ThreadLocalBase&) = delete;
+ ThreadLocalBase& operator=(const ThreadLocalBase&) = delete;
};
// Maps a thread to a set of ThreadLocals that have values instantiated on that
@@ -1497,7 +1464,7 @@
virtual void Run() = 0;
};
- ThreadWithParamBase(Runnable *runnable, Notification* thread_can_start);
+ ThreadWithParamBase(Runnable* runnable, Notification* thread_can_start);
virtual ~ThreadWithParamBase();
private:
@@ -1511,30 +1478,26 @@
typedef void UserThreadFunc(T);
ThreadWithParam(UserThreadFunc* func, T param, Notification* thread_can_start)
- : ThreadWithParamBase(new RunnableImpl(func, param), thread_can_start) {
- }
+ : ThreadWithParamBase(new RunnableImpl(func, param), thread_can_start) {}
virtual ~ThreadWithParam() {}
private:
class RunnableImpl : public Runnable {
public:
- RunnableImpl(UserThreadFunc* func, T param)
- : func_(func),
- param_(param) {
- }
+ RunnableImpl(UserThreadFunc* func, T param) : func_(func), param_(param) {}
virtual ~RunnableImpl() {}
- virtual void Run() {
- func_(param_);
- }
+ virtual void Run() { func_(param_); }
private:
UserThreadFunc* const func_;
const T param_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(RunnableImpl);
+ RunnableImpl(const RunnableImpl&) = delete;
+ RunnableImpl& operator=(const RunnableImpl&) = delete;
};
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ThreadWithParam);
+ ThreadWithParam(const ThreadWithParam&) = delete;
+ ThreadWithParam& operator=(const ThreadWithParam&) = delete;
};
// Implements thread-local storage on Windows systems.
@@ -1571,7 +1534,7 @@
explicit ThreadLocal(const T& value)
: default_factory_(new InstanceValueHolderFactory(value)) {}
- ~ThreadLocal() { ThreadLocalRegistry::OnThreadLocalDestroyed(this); }
+ ~ThreadLocal() override { ThreadLocalRegistry::OnThreadLocalDestroyed(this); }
T* pointer() { return GetOrCreateValue(); }
const T* pointer() const { return GetOrCreateValue(); }
@@ -1590,16 +1553,17 @@
private:
T value_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ValueHolder);
+ ValueHolder(const ValueHolder&) = delete;
+ ValueHolder& operator=(const ValueHolder&) = delete;
};
-
T* GetOrCreateValue() const {
return static_cast<ValueHolder*>(
- ThreadLocalRegistry::GetValueOnCurrentThread(this))->pointer();
+ ThreadLocalRegistry::GetValueOnCurrentThread(this))
+ ->pointer();
}
- virtual ThreadLocalValueHolderBase* NewValueForCurrentThread() const {
+ ThreadLocalValueHolderBase* NewValueForCurrentThread() const override {
return default_factory_->MakeNewHolder();
}
@@ -1610,7 +1574,8 @@
virtual ValueHolder* MakeNewHolder() const = 0;
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ValueHolderFactory);
+ ValueHolderFactory(const ValueHolderFactory&) = delete;
+ ValueHolderFactory& operator=(const ValueHolderFactory&) = delete;
};
class DefaultValueHolderFactory : public ValueHolderFactory {
@@ -1619,7 +1584,9 @@
ValueHolder* MakeNewHolder() const override { return new ValueHolder(); }
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(DefaultValueHolderFactory);
+ DefaultValueHolderFactory(const DefaultValueHolderFactory&) = delete;
+ DefaultValueHolderFactory& operator=(const DefaultValueHolderFactory&) =
+ delete;
};
class InstanceValueHolderFactory : public ValueHolderFactory {
@@ -1632,15 +1599,18 @@
private:
const T value_; // The value for each thread.
- GTEST_DISALLOW_COPY_AND_ASSIGN_(InstanceValueHolderFactory);
+ InstanceValueHolderFactory(const InstanceValueHolderFactory&) = delete;
+ InstanceValueHolderFactory& operator=(const InstanceValueHolderFactory&) =
+ delete;
};
std::unique_ptr<ValueHolderFactory> default_factory_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ThreadLocal);
+ ThreadLocal(const ThreadLocal&) = delete;
+ ThreadLocal& operator=(const ThreadLocal&) = delete;
};
-# elif GTEST_HAS_PTHREAD
+#elif GTEST_HAS_PTHREAD
// MutexBase and Mutex implement mutex on pthreads-based platforms.
class MutexBase {
@@ -1687,8 +1657,8 @@
};
// Forward-declares a static mutex.
-# define GTEST_DECLARE_STATIC_MUTEX_(mutex) \
- extern ::testing::internal::MutexBase mutex
+#define GTEST_DECLARE_STATIC_MUTEX_(mutex) \
+ extern ::testing::internal::MutexBase mutex
// Defines and statically (i.e. at link time) initializes a static mutex.
// The initialization list here does not explicitly initialize each field,
@@ -1707,12 +1677,11 @@
GTEST_CHECK_POSIX_SUCCESS_(pthread_mutex_init(&mutex_, nullptr));
has_owner_ = false;
}
- ~Mutex() {
- GTEST_CHECK_POSIX_SUCCESS_(pthread_mutex_destroy(&mutex_));
- }
+ ~Mutex() { GTEST_CHECK_POSIX_SUCCESS_(pthread_mutex_destroy(&mutex_)); }
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(Mutex);
+ Mutex(const Mutex&) = delete;
+ Mutex& operator=(const Mutex&) = delete;
};
// We cannot name this class MutexLock because the ctor declaration would
@@ -1722,15 +1691,15 @@
// "MutexLock l(&mu)". Hence the typedef trick below.
class GTestMutexLock {
public:
- explicit GTestMutexLock(MutexBase* mutex)
- : mutex_(mutex) { mutex_->Lock(); }
+ explicit GTestMutexLock(MutexBase* mutex) : mutex_(mutex) { mutex_->Lock(); }
~GTestMutexLock() { mutex_->Unlock(); }
private:
MutexBase* const mutex_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(GTestMutexLock);
+ GTestMutexLock(const GTestMutexLock&) = delete;
+ GTestMutexLock& operator=(const GTestMutexLock&) = delete;
};
typedef GTestMutexLock MutexLock;
@@ -1787,7 +1756,8 @@
private:
T value_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ValueHolder);
+ ValueHolder(const ValueHolder&) = delete;
+ ValueHolder& operator=(const ValueHolder&) = delete;
};
static pthread_key_t CreateKey() {
@@ -1819,7 +1789,8 @@
virtual ValueHolder* MakeNewHolder() const = 0;
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ValueHolderFactory);
+ ValueHolderFactory(const ValueHolderFactory&) = delete;
+ ValueHolderFactory& operator=(const ValueHolderFactory&) = delete;
};
class DefaultValueHolderFactory : public ValueHolderFactory {
@@ -1828,7 +1799,9 @@
ValueHolder* MakeNewHolder() const override { return new ValueHolder(); }
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(DefaultValueHolderFactory);
+ DefaultValueHolderFactory(const DefaultValueHolderFactory&) = delete;
+ DefaultValueHolderFactory& operator=(const DefaultValueHolderFactory&) =
+ delete;
};
class InstanceValueHolderFactory : public ValueHolderFactory {
@@ -1841,17 +1814,20 @@
private:
const T value_; // The value for each thread.
- GTEST_DISALLOW_COPY_AND_ASSIGN_(InstanceValueHolderFactory);
+ InstanceValueHolderFactory(const InstanceValueHolderFactory&) = delete;
+ InstanceValueHolderFactory& operator=(const InstanceValueHolderFactory&) =
+ delete;
};
// A key pthreads uses for looking up per-thread values.
const pthread_key_t key_;
std::unique_ptr<ValueHolderFactory> default_factory_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ThreadLocal);
+ ThreadLocal(const ThreadLocal&) = delete;
+ ThreadLocal& operator=(const ThreadLocal&) = delete;
};
-# endif // GTEST_HAS_MUTEX_AND_THREAD_LOCAL_
+#endif // GTEST_HAS_MUTEX_AND_THREAD_LOCAL_
#else // GTEST_IS_THREADSAFE
@@ -1868,10 +1844,10 @@
void AssertHeld() const {}
};
-# define GTEST_DECLARE_STATIC_MUTEX_(mutex) \
+#define GTEST_DECLARE_STATIC_MUTEX_(mutex) \
extern ::testing::internal::Mutex mutex
-# define GTEST_DEFINE_STATIC_MUTEX_(mutex) ::testing::internal::Mutex mutex
+#define GTEST_DEFINE_STATIC_MUTEX_(mutex) ::testing::internal::Mutex mutex
// We cannot name this class MutexLock because the ctor declaration would
// conflict with a macro named MutexLock, which is defined on some
@@ -1894,6 +1870,7 @@
const T* pointer() const { return &value_; }
const T& get() const { return value_; }
void set(const T& value) { value_ = value; }
+
private:
T value_;
};
@@ -1905,11 +1882,11 @@
GTEST_API_ size_t GetThreadCount();
#if GTEST_OS_WINDOWS
-# define GTEST_PATH_SEP_ "\\"
-# define GTEST_HAS_ALT_PATH_SEP_ 1
+#define GTEST_PATH_SEP_ "\\"
+#define GTEST_HAS_ALT_PATH_SEP_ 1
#else
-# define GTEST_PATH_SEP_ "/"
-# define GTEST_HAS_ALT_PATH_SEP_ 0
+#define GTEST_PATH_SEP_ "/"
+#define GTEST_HAS_ALT_PATH_SEP_ 0
#endif // GTEST_OS_WINDOWS
// Utilities for char.
@@ -1967,8 +1944,7 @@
inline std::string StripTrailingSpaces(std::string str) {
std::string::iterator it = str.end();
- while (it != str.begin() && IsSpace(*--it))
- it = str.erase(it);
+ while (it != str.begin() && IsSpace(*--it)) it = str.erase(it);
return str;
}
@@ -1986,36 +1962,35 @@
typedef struct _stat StatStruct;
-# ifdef __BORLANDC__
+#ifdef __BORLANDC__
inline int DoIsATTY(int fd) { return isatty(fd); }
inline int StrCaseCmp(const char* s1, const char* s2) {
return stricmp(s1, s2);
}
inline char* StrDup(const char* src) { return strdup(src); }
-# else // !__BORLANDC__
-# if GTEST_OS_WINDOWS_MOBILE
+#else // !__BORLANDC__
+#if GTEST_OS_WINDOWS_MOBILE || GTEST_OS_ZOS || GTEST_OS_IOS || \
+ GTEST_OS_WINDOWS_PHONE || GTEST_OS_WINDOWS_RT || defined(ESP_PLATFORM)
inline int DoIsATTY(int /* fd */) { return 0; }
-# else
+#else
inline int DoIsATTY(int fd) { return _isatty(fd); }
-# endif // GTEST_OS_WINDOWS_MOBILE
+#endif // GTEST_OS_WINDOWS_MOBILE
inline int StrCaseCmp(const char* s1, const char* s2) {
return _stricmp(s1, s2);
}
inline char* StrDup(const char* src) { return _strdup(src); }
-# endif // __BORLANDC__
+#endif // __BORLANDC__
-# if GTEST_OS_WINDOWS_MOBILE
+#if GTEST_OS_WINDOWS_MOBILE
inline int FileNo(FILE* file) { return reinterpret_cast<int>(_fileno(file)); }
// Stat(), RmDir(), and IsDir() are not needed on Windows CE at this
// time and thus not defined there.
-# else
+#else
inline int FileNo(FILE* file) { return _fileno(file); }
inline int Stat(const char* path, StatStruct* buf) { return _stat(path, buf); }
inline int RmDir(const char* dir) { return _rmdir(dir); }
-inline bool IsDir(const StatStruct& st) {
- return (_S_IFDIR & st.st_mode) != 0;
-}
-# endif // GTEST_OS_WINDOWS_MOBILE
+inline bool IsDir(const StatStruct& st) { return (_S_IFDIR & st.st_mode) != 0; }
+#endif // GTEST_OS_WINDOWS_MOBILE
#elif GTEST_OS_ESP8266
typedef struct stat StatStruct;
@@ -2079,12 +2054,12 @@
std::wstring wide_path = converter.from_bytes(path);
std::wstring wide_mode = converter.from_bytes(mode);
return _wfopen(wide_path.c_str(), wide_mode.c_str());
-#else // GTEST_OS_WINDOWS && !GTEST_OS_WINDOWS_MINGW
+#else // GTEST_OS_WINDOWS && !GTEST_OS_WINDOWS_MINGW
return fopen(path, mode);
#endif // GTEST_OS_WINDOWS && !GTEST_OS_WINDOWS_MINGW
}
#if !GTEST_OS_WINDOWS_MOBILE
-inline FILE *FReopen(const char* path, const char* mode, FILE* stream) {
+inline FILE* FReopen(const char* path, const char* mode, FILE* stream) {
return freopen(path, mode, stream);
}
inline FILE* FDOpen(int fd, const char* mode) { return fdopen(fd, mode); }
@@ -2136,13 +2111,13 @@
// snprintf is a variadic function.
#if _MSC_VER && !GTEST_OS_WINDOWS_MOBILE
// MSVC 2005 and above support variadic macros.
-# define GTEST_SNPRINTF_(buffer, size, format, ...) \
- _snprintf_s(buffer, size, size, format, __VA_ARGS__)
+#define GTEST_SNPRINTF_(buffer, size, format, ...) \
+ _snprintf_s(buffer, size, size, format, __VA_ARGS__)
#elif defined(_MSC_VER)
// Windows CE does not define _snprintf_s
-# define GTEST_SNPRINTF_ _snprintf
+#define GTEST_SNPRINTF_ _snprintf
#else
-# define GTEST_SNPRINTF_ snprintf
+#define GTEST_SNPRINTF_ snprintf
#endif
// The biggest signed integer type the compiler supports.
@@ -2202,37 +2177,84 @@
// Macro for referencing flags.
#if !defined(GTEST_FLAG)
-# define GTEST_FLAG(name) FLAGS_gtest_##name
+#define GTEST_FLAG_NAME_(name) gtest_##name
+#define GTEST_FLAG(name) FLAGS_gtest_##name
#endif // !defined(GTEST_FLAG)
-#if !defined(GTEST_USE_OWN_FLAGFILE_FLAG_)
-# define GTEST_USE_OWN_FLAGFILE_FLAG_ 1
-#endif // !defined(GTEST_USE_OWN_FLAGFILE_FLAG_)
-
-#if !defined(GTEST_DECLARE_bool_)
-# define GTEST_FLAG_SAVER_ ::testing::internal::GTestFlagSaver
-
-// Macros for declaring flags.
-# define GTEST_DECLARE_bool_(name) GTEST_API_ extern bool GTEST_FLAG(name)
-# define GTEST_DECLARE_int32_(name) \
- GTEST_API_ extern std::int32_t GTEST_FLAG(name)
-# define GTEST_DECLARE_string_(name) \
- GTEST_API_ extern ::std::string GTEST_FLAG(name)
+// Pick a command line flags implementation.
+#if GTEST_HAS_ABSL
// Macros for defining flags.
-# define GTEST_DEFINE_bool_(name, default_val, doc) \
- GTEST_API_ bool GTEST_FLAG(name) = (default_val)
-# define GTEST_DEFINE_int32_(name, default_val, doc) \
- GTEST_API_ std::int32_t GTEST_FLAG(name) = (default_val)
-# define GTEST_DEFINE_string_(name, default_val, doc) \
- GTEST_API_ ::std::string GTEST_FLAG(name) = (default_val)
+#define GTEST_DEFINE_bool_(name, default_val, doc) \
+ ABSL_FLAG(bool, GTEST_FLAG_NAME_(name), default_val, doc)
+#define GTEST_DEFINE_int32_(name, default_val, doc) \
+ ABSL_FLAG(int32_t, GTEST_FLAG_NAME_(name), default_val, doc)
+#define GTEST_DEFINE_string_(name, default_val, doc) \
+ ABSL_FLAG(std::string, GTEST_FLAG_NAME_(name), default_val, doc)
-#endif // !defined(GTEST_DECLARE_bool_)
+// Macros for declaring flags.
+#define GTEST_DECLARE_bool_(name) \
+ ABSL_DECLARE_FLAG(bool, GTEST_FLAG_NAME_(name))
+#define GTEST_DECLARE_int32_(name) \
+ ABSL_DECLARE_FLAG(int32_t, GTEST_FLAG_NAME_(name))
+#define GTEST_DECLARE_string_(name) \
+ ABSL_DECLARE_FLAG(std::string, GTEST_FLAG_NAME_(name))
+
+#define GTEST_FLAG_SAVER_ ::absl::FlagSaver
+
+#define GTEST_FLAG_GET(name) ::absl::GetFlag(GTEST_FLAG(name))
+#define GTEST_FLAG_SET(name, value) \
+ (void)(::absl::SetFlag(>EST_FLAG(name), value))
+#define GTEST_USE_OWN_FLAGFILE_FLAG_ 0
+
+#else // GTEST_HAS_ABSL
+
+// Macros for defining flags.
+#define GTEST_DEFINE_bool_(name, default_val, doc) \
+ namespace testing { \
+ GTEST_API_ bool GTEST_FLAG(name) = (default_val); \
+ } \
+ static_assert(true, "no-op to require trailing semicolon")
+#define GTEST_DEFINE_int32_(name, default_val, doc) \
+ namespace testing { \
+ GTEST_API_ std::int32_t GTEST_FLAG(name) = (default_val); \
+ } \
+ static_assert(true, "no-op to require trailing semicolon")
+#define GTEST_DEFINE_string_(name, default_val, doc) \
+ namespace testing { \
+ GTEST_API_ ::std::string GTEST_FLAG(name) = (default_val); \
+ } \
+ static_assert(true, "no-op to require trailing semicolon")
+
+// Macros for declaring flags.
+#define GTEST_DECLARE_bool_(name) \
+ namespace testing { \
+ GTEST_API_ extern bool GTEST_FLAG(name); \
+ } \
+ static_assert(true, "no-op to require trailing semicolon")
+#define GTEST_DECLARE_int32_(name) \
+ namespace testing { \
+ GTEST_API_ extern std::int32_t GTEST_FLAG(name); \
+ } \
+ static_assert(true, "no-op to require trailing semicolon")
+#define GTEST_DECLARE_string_(name) \
+ namespace testing { \
+ GTEST_API_ extern ::std::string GTEST_FLAG(name); \
+ } \
+ static_assert(true, "no-op to require trailing semicolon")
+
+#define GTEST_FLAG_SAVER_ ::testing::internal::GTestFlagSaver
+
+#define GTEST_FLAG_GET(name) ::testing::GTEST_FLAG(name)
+#define GTEST_FLAG_SET(name, value) (void)(::testing::GTEST_FLAG(name) = value)
+#define GTEST_USE_OWN_FLAGFILE_FLAG_ 1
+
+#endif // GTEST_HAS_ABSL
// Thread annotations
#if !defined(GTEST_EXCLUSIVE_LOCK_REQUIRED_)
-# define GTEST_EXCLUSIVE_LOCK_REQUIRED_(locks)
-# define GTEST_LOCK_EXCLUDED_(locks)
+#define GTEST_EXCLUSIVE_LOCK_REQUIRED_(locks)
+#define GTEST_LOCK_EXCLUDED_(locks)
#endif // !defined(GTEST_EXCLUSIVE_LOCK_REQUIRED_)
// Parses 'str' for a 32-bit signed integer. If successful, writes the result
@@ -2308,6 +2330,7 @@
namespace internal {
template <typename T>
using Optional = ::absl::optional<T>;
+inline ::absl::nullopt_t Nullopt() { return ::absl::nullopt; }
} // namespace internal
} // namespace testing
#else
@@ -2321,6 +2344,7 @@
namespace internal {
template <typename T>
using Optional = ::std::optional<T>;
+inline ::std::nullopt_t Nullopt() { return ::std::nullopt; }
} // namespace internal
} // namespace testing
// The case where absl is configured NOT to alias std::optional is not
@@ -2332,7 +2356,7 @@
#if GTEST_HAS_ABSL
// Always use absl::string_view for Matcher<> specializations if googletest
// is built with absl support.
-# define GTEST_INTERNAL_HAS_STRING_VIEW 1
+#define GTEST_INTERNAL_HAS_STRING_VIEW 1
#include "absl/strings/string_view.h"
namespace testing {
namespace internal {
@@ -2340,11 +2364,11 @@
} // namespace internal
} // namespace testing
#else
-# ifdef __has_include
-# if __has_include(<string_view>) && __cplusplus >= 201703L
+#ifdef __has_include
+#if __has_include(<string_view>) && __cplusplus >= 201703L
// Otherwise for C++17 and higher use std::string_view for Matcher<>
// specializations.
-# define GTEST_INTERNAL_HAS_STRING_VIEW 1
+#define GTEST_INTERNAL_HAS_STRING_VIEW 1
#include <string_view>
namespace testing {
namespace internal {
@@ -2353,8 +2377,8 @@
} // namespace testing
// The case where absl is configured NOT to alias std::string_view is not
// supported.
-# endif // __has_include(<string_view>) && __cplusplus >= 201703L
-# endif // __has_include
+#endif // __has_include(<string_view>) && __cplusplus >= 201703L
+#endif // __has_include
#endif // GTEST_HAS_ABSL
#if GTEST_HAS_ABSL
diff --git a/third_party/googletest/src/googletest/include/gtest/internal/gtest-string.h b/third_party/googletest/src/googletest/include/gtest/internal/gtest-string.h
index 10f774f..cca2e1f 100644
--- a/third_party/googletest/src/googletest/include/gtest/internal/gtest-string.h
+++ b/third_party/googletest/src/googletest/include/gtest/internal/gtest-string.h
@@ -26,7 +26,7 @@
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-//
+
// The Google C++ Testing and Mocking Framework (Google Test)
//
// This header file declares the String class and functions used internally by
@@ -36,17 +36,20 @@
// This header file is #included by gtest-internal.h.
// It should not be #included by other files.
-// GOOGLETEST_CM0001 DO NOT DELETE
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_STRING_H_
#define GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_STRING_H_
#ifdef __BORLANDC__
// string.h is not guaranteed to provide strcpy on C++ Builder.
-# include <mem.h>
+#include <mem.h>
#endif
#include <string.h>
+
#include <cstdint>
#include <string>
@@ -123,8 +126,7 @@
// Unlike strcasecmp(), this function can handle NULL argument(s).
// A NULL C string is considered different to any non-NULL C string,
// including the empty string.
- static bool CaseInsensitiveCStringEquals(const char* lhs,
- const char* rhs);
+ static bool CaseInsensitiveCStringEquals(const char* lhs, const char* rhs);
// Compares two wide C strings, ignoring case. Returns true if and only if
// they have the same content.
@@ -143,8 +145,8 @@
// Returns true if and only if the given string ends with the given suffix,
// ignoring case. Any string is considered to end with an empty suffix.
- static bool EndsWithCaseInsensitive(
- const std::string& str, const std::string& suffix);
+ static bool EndsWithCaseInsensitive(const std::string& str,
+ const std::string& suffix);
// Formats an int value as "%02d".
static std::string FormatIntWidth2(int value); // "%02d" for width == 2
@@ -163,7 +165,7 @@
private:
String(); // Not meant to be instantiated.
-}; // class String
+}; // class String
// Gets the content of the stringstream's buffer as an std::string. Each '\0'
// character in the buffer is replaced with "\\0".
diff --git a/third_party/googletest/src/googletest/include/gtest/internal/gtest-type-util.h b/third_party/googletest/src/googletest/include/gtest/internal/gtest-type-util.h
index b87a2e2..6bc02a7 100644
--- a/third_party/googletest/src/googletest/include/gtest/internal/gtest-type-util.h
+++ b/third_party/googletest/src/googletest/include/gtest/internal/gtest-type-util.h
@@ -30,7 +30,9 @@
// Type utilities needed for implementing typed and type-parameterized
// tests.
-// GOOGLETEST_CM0001 DO NOT DELETE
+// IWYU pragma: private, include "gtest/gtest.h"
+// IWYU pragma: friend gtest/.*
+// IWYU pragma: friend gmock/.*
#ifndef GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_TYPE_UTIL_H_
#define GOOGLETEST_INCLUDE_GTEST_INTERNAL_GTEST_TYPE_UTIL_H_
@@ -39,11 +41,11 @@
// #ifdef __GNUC__ is too general here. It is possible to use gcc without using
// libstdc++ (which is where cxxabi.h comes from).
-# if GTEST_HAS_CXXABI_H_
-# include <cxxabi.h>
-# elif defined(__HP_aCC)
-# include <acxx_demangle.h>
-# endif // GTEST_HASH_CXXABI_H_
+#if GTEST_HAS_CXXABI_H_
+#include <cxxabi.h>
+#elif defined(__HP_aCC)
+#include <acxx_demangle.h>
+#endif // GTEST_HASH_CXXABI_H_
namespace testing {
namespace internal {
@@ -101,7 +103,9 @@
// A unique type indicating an empty node
struct None {};
-# define GTEST_TEMPLATE_ template <typename T> class
+#define GTEST_TEMPLATE_ \
+ template <typename T> \
+ class
// The template "selector" struct TemplateSel<Tmpl> is used to
// represent Tmpl, which must be a class template with one type
@@ -119,8 +123,7 @@
};
};
-# define GTEST_BIND_(TmplSel, T) \
- TmplSel::template Bind<T>::type
+#define GTEST_BIND_(TmplSel, T) TmplSel::template Bind<T>::type
template <GTEST_TEMPLATE_ Head_, GTEST_TEMPLATE_... Tail_>
struct Templates {
diff --git a/third_party/googletest/src/googletest/src/gtest-all.cc b/third_party/googletest/src/googletest/src/gtest-all.cc
index ad29290..2a70ed8 100644
--- a/third_party/googletest/src/googletest/src/gtest-all.cc
+++ b/third_party/googletest/src/googletest/src/gtest-all.cc
@@ -38,7 +38,7 @@
#include "gtest/gtest.h"
// The following lines pull in the real gtest *.cc files.
-#include "src/gtest.cc"
+#include "src/gtest-assertion-result.cc"
#include "src/gtest-death-test.cc"
#include "src/gtest-filepath.cc"
#include "src/gtest-matchers.cc"
@@ -46,3 +46,4 @@
#include "src/gtest-printers.cc"
#include "src/gtest-test-part.cc"
#include "src/gtest-typed-test.cc"
+#include "src/gtest.cc"
diff --git a/third_party/googletest/src/googletest/src/gtest-assertion-result.cc b/third_party/googletest/src/googletest/src/gtest-assertion-result.cc
new file mode 100644
index 0000000..f1c0b10
--- /dev/null
+++ b/third_party/googletest/src/googletest/src/gtest-assertion-result.cc
@@ -0,0 +1,77 @@
+// Copyright 2005, Google Inc.
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are
+// met:
+//
+// * Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// * Redistributions in binary form must reproduce the above
+// copyright notice, this list of conditions and the following disclaimer
+// in the documentation and/or other materials provided with the
+// distribution.
+// * Neither the name of Google Inc. nor the names of its
+// contributors may be used to endorse or promote products derived from
+// this software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+// The Google C++ Testing and Mocking Framework (Google Test)
+//
+// This file defines the AssertionResult type.
+
+#include "gtest/gtest-assertion-result.h"
+
+#include <string>
+#include <utility>
+
+#include "gtest/gtest-message.h"
+
+namespace testing {
+
+// AssertionResult constructors.
+// Used in EXPECT_TRUE/FALSE(assertion_result).
+AssertionResult::AssertionResult(const AssertionResult& other)
+ : success_(other.success_),
+ message_(other.message_.get() != nullptr
+ ? new ::std::string(*other.message_)
+ : static_cast< ::std::string*>(nullptr)) {}
+
+// Swaps two AssertionResults.
+void AssertionResult::swap(AssertionResult& other) {
+ using std::swap;
+ swap(success_, other.success_);
+ swap(message_, other.message_);
+}
+
+// Returns the assertion's negation. Used with EXPECT/ASSERT_FALSE.
+AssertionResult AssertionResult::operator!() const {
+ AssertionResult negation(!success_);
+ if (message_.get() != nullptr) negation << *message_;
+ return negation;
+}
+
+// Makes a successful assertion result.
+AssertionResult AssertionSuccess() { return AssertionResult(true); }
+
+// Makes a failed assertion result.
+AssertionResult AssertionFailure() { return AssertionResult(false); }
+
+// Makes a failed assertion result with the given failure message.
+// Deprecated; use AssertionFailure() << message.
+AssertionResult AssertionFailure(const Message& message) {
+ return AssertionFailure() << message;
+}
+
+} // namespace testing
diff --git a/third_party/googletest/src/googletest/src/gtest-death-test.cc b/third_party/googletest/src/googletest/src/gtest-death-test.cc
index bf4f633..e6abc62 100644
--- a/third_party/googletest/src/googletest/src/gtest-death-test.cc
+++ b/third_party/googletest/src/googletest/src/gtest-death-test.cc
@@ -35,49 +35,49 @@
#include <functional>
#include <utility>
-#include "gtest/internal/gtest-port.h"
#include "gtest/internal/custom/gtest.h"
+#include "gtest/internal/gtest-port.h"
#if GTEST_HAS_DEATH_TEST
-# if GTEST_OS_MAC
-# include <crt_externs.h>
-# endif // GTEST_OS_MAC
+#if GTEST_OS_MAC
+#include <crt_externs.h>
+#endif // GTEST_OS_MAC
-# include <errno.h>
-# include <fcntl.h>
-# include <limits.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
-# if GTEST_OS_LINUX
-# include <signal.h>
-# endif // GTEST_OS_LINUX
+#if GTEST_OS_LINUX
+#include <signal.h>
+#endif // GTEST_OS_LINUX
-# include <stdarg.h>
+#include <stdarg.h>
-# if GTEST_OS_WINDOWS
-# include <windows.h>
-# else
-# include <sys/mman.h>
-# include <sys/wait.h>
-# endif // GTEST_OS_WINDOWS
+#if GTEST_OS_WINDOWS
+#include <windows.h>
+#else
+#include <sys/mman.h>
+#include <sys/wait.h>
+#endif // GTEST_OS_WINDOWS
-# if GTEST_OS_QNX
-# include <spawn.h>
-# endif // GTEST_OS_QNX
+#if GTEST_OS_QNX
+#include <spawn.h>
+#endif // GTEST_OS_QNX
-# if GTEST_OS_FUCHSIA
-# include <lib/fdio/fd.h>
-# include <lib/fdio/io.h>
-# include <lib/fdio/spawn.h>
-# include <lib/zx/channel.h>
-# include <lib/zx/port.h>
-# include <lib/zx/process.h>
-# include <lib/zx/socket.h>
-# include <zircon/processargs.h>
-# include <zircon/syscalls.h>
-# include <zircon/syscalls/policy.h>
-# include <zircon/syscalls/port.h>
-# endif // GTEST_OS_FUCHSIA
+#if GTEST_OS_FUCHSIA
+#include <lib/fdio/fd.h>
+#include <lib/fdio/io.h>
+#include <lib/fdio/spawn.h>
+#include <lib/zx/channel.h>
+#include <lib/zx/port.h>
+#include <lib/zx/process.h>
+#include <lib/zx/socket.h>
+#include <zircon/processargs.h>
+#include <zircon/syscalls.h>
+#include <zircon/syscalls/policy.h>
+#include <zircon/syscalls/port.h>
+#endif // GTEST_OS_FUCHSIA
#endif // GTEST_HAS_DEATH_TEST
@@ -96,9 +96,12 @@
// used internally at Google, is "threadsafe".
static const char kDefaultDeathTestStyle[] = GTEST_DEFAULT_DEATH_TEST_STYLE;
+} // namespace testing
+
GTEST_DEFINE_string_(
death_test_style,
- internal::StringFromGTestEnv("death_test_style", kDefaultDeathTestStyle),
+ testing::internal::StringFromGTestEnv("death_test_style",
+ testing::kDefaultDeathTestStyle),
"Indicates how to run a death test in a forked child process: "
"\"threadsafe\" (child process re-executes the test binary "
"from the beginning, running only the specific death test) or "
@@ -107,7 +110,7 @@
GTEST_DEFINE_bool_(
death_test_use_fork,
- internal::BoolFromGTestEnv("death_test_use_fork", false),
+ testing::internal::BoolFromGTestEnv("death_test_use_fork", false),
"Instructs to use fork()/_exit() instead of clone() in death tests. "
"Ignored and always uses fork() on POSIX systems where clone() is not "
"implemented. Useful when running under valgrind or similar tools if "
@@ -117,7 +120,6 @@
"work in 99% of the cases. Once valgrind is fixed, this flag will "
"most likely be removed.");
-namespace internal {
GTEST_DEFINE_string_(
internal_run_death_test, "",
"Indicates the file, line number, temporal index of "
@@ -126,7 +128,8 @@
"the '|' characters. This flag is specified if and only if the "
"current process is a sub-process launched for running a thread-safe "
"death test. FOR INTERNAL USE ONLY.");
-} // namespace internal
+
+namespace testing {
#if GTEST_HAS_DEATH_TEST
@@ -134,9 +137,9 @@
// Valid only for fast death tests. Indicates the code is running in the
// child process of a fast style death test.
-# if !GTEST_OS_WINDOWS && !GTEST_OS_FUCHSIA
+#if !GTEST_OS_WINDOWS && !GTEST_OS_FUCHSIA
static bool g_in_fast_death_test_child = false;
-# endif
+#endif
// Returns a Boolean value indicating whether the caller is currently
// executing in the context of the death test child process. Tools such as
@@ -144,16 +147,16 @@
// tests. IMPORTANT: This is an internal utility. Using it may break the
// implementation of death tests. User code MUST NOT use it.
bool InDeathTestChild() {
-# if GTEST_OS_WINDOWS || GTEST_OS_FUCHSIA
+#if GTEST_OS_WINDOWS || GTEST_OS_FUCHSIA
// On Windows and Fuchsia, death tests are thread-safe regardless of the value
// of the death_test_style flag.
- return !GTEST_FLAG(internal_run_death_test).empty();
+ return !GTEST_FLAG_GET(internal_run_death_test).empty();
-# else
+#else
- if (GTEST_FLAG(death_test_style) == "threadsafe")
- return !GTEST_FLAG(internal_run_death_test).empty();
+ if (GTEST_FLAG_GET(death_test_style) == "threadsafe")
+ return !GTEST_FLAG_GET(internal_run_death_test).empty();
else
return g_in_fast_death_test_child;
#endif
@@ -162,40 +165,38 @@
} // namespace internal
// ExitedWithCode constructor.
-ExitedWithCode::ExitedWithCode(int exit_code) : exit_code_(exit_code) {
-}
+ExitedWithCode::ExitedWithCode(int exit_code) : exit_code_(exit_code) {}
// ExitedWithCode function-call operator.
bool ExitedWithCode::operator()(int exit_status) const {
-# if GTEST_OS_WINDOWS || GTEST_OS_FUCHSIA
+#if GTEST_OS_WINDOWS || GTEST_OS_FUCHSIA
return exit_status == exit_code_;
-# else
+#else
return WIFEXITED(exit_status) && WEXITSTATUS(exit_status) == exit_code_;
-# endif // GTEST_OS_WINDOWS || GTEST_OS_FUCHSIA
+#endif // GTEST_OS_WINDOWS || GTEST_OS_FUCHSIA
}
-# if !GTEST_OS_WINDOWS && !GTEST_OS_FUCHSIA
+#if !GTEST_OS_WINDOWS && !GTEST_OS_FUCHSIA
// KilledBySignal constructor.
-KilledBySignal::KilledBySignal(int signum) : signum_(signum) {
-}
+KilledBySignal::KilledBySignal(int signum) : signum_(signum) {}
// KilledBySignal function-call operator.
bool KilledBySignal::operator()(int exit_status) const {
-# if defined(GTEST_KILLED_BY_SIGNAL_OVERRIDE_)
+#if defined(GTEST_KILLED_BY_SIGNAL_OVERRIDE_)
{
bool result;
if (GTEST_KILLED_BY_SIGNAL_OVERRIDE_(signum_, exit_status, &result)) {
return result;
}
}
-# endif // defined(GTEST_KILLED_BY_SIGNAL_OVERRIDE_)
+#endif // defined(GTEST_KILLED_BY_SIGNAL_OVERRIDE_)
return WIFSIGNALED(exit_status) && WTERMSIG(exit_status) == signum_;
}
-# endif // !GTEST_OS_WINDOWS && !GTEST_OS_FUCHSIA
+#endif // !GTEST_OS_WINDOWS && !GTEST_OS_FUCHSIA
namespace internal {
@@ -206,23 +207,23 @@
static std::string ExitSummary(int exit_code) {
Message m;
-# if GTEST_OS_WINDOWS || GTEST_OS_FUCHSIA
+#if GTEST_OS_WINDOWS || GTEST_OS_FUCHSIA
m << "Exited with exit status " << exit_code;
-# else
+#else
if (WIFEXITED(exit_code)) {
m << "Exited with exit status " << WEXITSTATUS(exit_code);
} else if (WIFSIGNALED(exit_code)) {
m << "Terminated by signal " << WTERMSIG(exit_code);
}
-# ifdef WCOREDUMP
+#ifdef WCOREDUMP
if (WCOREDUMP(exit_code)) {
m << " (core dumped)";
}
-# endif
-# endif // GTEST_OS_WINDOWS || GTEST_OS_FUCHSIA
+#endif
+#endif // GTEST_OS_WINDOWS || GTEST_OS_FUCHSIA
return m.GetString();
}
@@ -233,7 +234,7 @@
return !ExitedWithCode(0)(exit_status);
}
-# if !GTEST_OS_WINDOWS && !GTEST_OS_FUCHSIA
+#if !GTEST_OS_WINDOWS && !GTEST_OS_FUCHSIA
// Generates a textual failure message when a death test finds more than
// one thread running, or cannot determine the number of threads, prior
// to executing the given statement. It is the responsibility of the
@@ -254,7 +255,7 @@
<< " this is the last message you see before your test times out.";
return msg.GetString();
}
-# endif // !GTEST_OS_WINDOWS && !GTEST_OS_FUCHSIA
+#endif // !GTEST_OS_WINDOWS && !GTEST_OS_FUCHSIA
// Flag characters for reporting a death test that did not die.
static const char kDeathTestLived = 'L';
@@ -304,14 +305,14 @@
// A replacement for CHECK that calls DeathTestAbort if the assertion
// fails.
-# define GTEST_DEATH_TEST_CHECK_(expression) \
- do { \
- if (!::testing::internal::IsTrue(expression)) { \
- DeathTestAbort( \
- ::std::string("CHECK failed: File ") + __FILE__ + ", line " \
- + ::testing::internal::StreamableToString(__LINE__) + ": " \
- + #expression); \
- } \
+#define GTEST_DEATH_TEST_CHECK_(expression) \
+ do { \
+ if (!::testing::internal::IsTrue(expression)) { \
+ DeathTestAbort(::std::string("CHECK failed: File ") + __FILE__ + \
+ ", line " + \
+ ::testing::internal::StreamableToString(__LINE__) + \
+ ": " + #expression); \
+ } \
} while (::testing::internal::AlwaysFalse())
// This macro is similar to GTEST_DEATH_TEST_CHECK_, but it is meant for
@@ -321,23 +322,23 @@
// evaluates the expression as long as it evaluates to -1 and sets
// errno to EINTR. If the expression evaluates to -1 but errno is
// something other than EINTR, DeathTestAbort is called.
-# define GTEST_DEATH_TEST_CHECK_SYSCALL_(expression) \
- do { \
- int gtest_retval; \
- do { \
- gtest_retval = (expression); \
- } while (gtest_retval == -1 && errno == EINTR); \
- if (gtest_retval == -1) { \
- DeathTestAbort( \
- ::std::string("CHECK failed: File ") + __FILE__ + ", line " \
- + ::testing::internal::StreamableToString(__LINE__) + ": " \
- + #expression + " != -1"); \
- } \
+#define GTEST_DEATH_TEST_CHECK_SYSCALL_(expression) \
+ do { \
+ int gtest_retval; \
+ do { \
+ gtest_retval = (expression); \
+ } while (gtest_retval == -1 && errno == EINTR); \
+ if (gtest_retval == -1) { \
+ DeathTestAbort(::std::string("CHECK failed: File ") + __FILE__ + \
+ ", line " + \
+ ::testing::internal::StreamableToString(__LINE__) + \
+ ": " + #expression + " != -1"); \
+ } \
} while (::testing::internal::AlwaysFalse())
// Returns the message describing the last system error in errno.
std::string GetLastErrnoDescription() {
- return errno == 0 ? "" : posix::StrError(errno);
+ return errno == 0 ? "" : posix::StrError(errno);
}
// This is called from a death test parent process to read a failure
@@ -370,8 +371,9 @@
DeathTest::DeathTest() {
TestInfo* const info = GetUnitTestImpl()->current_test_info();
if (info == nullptr) {
- DeathTestAbort("Cannot run a death test outside of a TEST or "
- "TEST_F construct");
+ DeathTestAbort(
+ "Cannot run a death test outside of a TEST or "
+ "TEST_F construct");
}
}
@@ -500,9 +502,7 @@
set_read_fd(-1);
}
-std::string DeathTestImpl::GetErrorLogs() {
- return GetCapturedStderr();
-}
+std::string DeathTestImpl::GetErrorLogs() { return GetCapturedStderr(); }
// Signals that the death test code which should have exited, didn't.
// Should be called only in a death test child process.
@@ -512,9 +512,9 @@
// The parent process considers the death test to be a failure if
// it finds any data in our pipe. So, here we write a single flag byte
// to the pipe, then exit.
- const char status_ch =
- reason == TEST_DID_NOT_DIE ? kDeathTestLived :
- reason == TEST_THREW_EXCEPTION ? kDeathTestThrew : kDeathTestReturned;
+ const char status_ch = reason == TEST_DID_NOT_DIE ? kDeathTestLived
+ : reason == TEST_THREW_EXCEPTION ? kDeathTestThrew
+ : kDeathTestReturned;
GTEST_DEATH_TEST_CHECK_SYSCALL_(posix::Write(write_fd(), &status_ch, 1));
// We are leaking the descriptor here because on some platforms (i.e.,
@@ -533,7 +533,7 @@
// much easier.
static ::std::string FormatDeathTestOutput(const ::std::string& output) {
::std::string ret;
- for (size_t at = 0; ; ) {
+ for (size_t at = 0;;) {
const size_t line_end = output.find('\n', at);
ret += "[ DEATH ] ";
if (line_end == ::std::string::npos) {
@@ -568,8 +568,7 @@
// the first failing condition, in the order given above, is the one that is
// reported. Also sets the last death test message string.
bool DeathTestImpl::Passed(bool status_ok) {
- if (!spawned())
- return false;
+ if (!spawned()) return false;
const std::string error_message = GetErrorLogs();
@@ -580,15 +579,18 @@
switch (outcome()) {
case LIVED:
buffer << " Result: failed to die.\n"
- << " Error msg:\n" << FormatDeathTestOutput(error_message);
+ << " Error msg:\n"
+ << FormatDeathTestOutput(error_message);
break;
case THREW:
buffer << " Result: threw an exception.\n"
- << " Error msg:\n" << FormatDeathTestOutput(error_message);
+ << " Error msg:\n"
+ << FormatDeathTestOutput(error_message);
break;
case RETURNED:
buffer << " Result: illegal return in test statement.\n"
- << " Error msg:\n" << FormatDeathTestOutput(error_message);
+ << " Error msg:\n"
+ << FormatDeathTestOutput(error_message);
break;
case DIED:
if (status_ok) {
@@ -605,7 +607,8 @@
} else {
buffer << " Result: died but not with expected exit code:\n"
<< " " << ExitSummary(status()) << "\n"
- << "Actual msg:\n" << FormatDeathTestOutput(error_message);
+ << "Actual msg:\n"
+ << FormatDeathTestOutput(error_message);
}
break;
case IN_PROGRESS:
@@ -618,7 +621,7 @@
return success;
}
-# if GTEST_OS_WINDOWS
+#if GTEST_OS_WINDOWS
// WindowsDeathTest implements death tests on Windows. Due to the
// specifics of starting new processes on Windows, death tests there are
// always threadsafe, and Google Test considers the
@@ -679,14 +682,12 @@
// status, or 0 if no child process exists. As a side effect, sets the
// outcome data member.
int WindowsDeathTest::Wait() {
- if (!spawned())
- return 0;
+ if (!spawned()) return 0;
// Wait until the child either signals that it has acquired the write end
// of the pipe or it dies.
- const HANDLE wait_handles[2] = { child_handle_.Get(), event_handle_.Get() };
- switch (::WaitForMultipleObjects(2,
- wait_handles,
+ const HANDLE wait_handles[2] = {child_handle_.Get(), event_handle_.Get()};
+ switch (::WaitForMultipleObjects(2, wait_handles,
FALSE, // Waits for any of the handles.
INFINITE)) {
case WAIT_OBJECT_0:
@@ -707,9 +708,8 @@
// returns immediately if the child has already exited, regardless of
// whether previous calls to WaitForMultipleObjects synchronized on this
// handle or not.
- GTEST_DEATH_TEST_CHECK_(
- WAIT_OBJECT_0 == ::WaitForSingleObject(child_handle_.Get(),
- INFINITE));
+ GTEST_DEATH_TEST_CHECK_(WAIT_OBJECT_0 ==
+ ::WaitForSingleObject(child_handle_.Get(), INFINITE));
DWORD status_code;
GTEST_DEATH_TEST_CHECK_(
::GetExitCodeProcess(child_handle_.Get(), &status_code) != FALSE);
@@ -742,12 +742,12 @@
SECURITY_ATTRIBUTES handles_are_inheritable = {sizeof(SECURITY_ATTRIBUTES),
nullptr, TRUE};
HANDLE read_handle, write_handle;
- GTEST_DEATH_TEST_CHECK_(
- ::CreatePipe(&read_handle, &write_handle, &handles_are_inheritable,
- 0) // Default buffer size.
- != FALSE);
- set_read_fd(::_open_osfhandle(reinterpret_cast<intptr_t>(read_handle),
- O_RDONLY));
+ GTEST_DEATH_TEST_CHECK_(::CreatePipe(&read_handle, &write_handle,
+ &handles_are_inheritable,
+ 0) // Default buffer size.
+ != FALSE);
+ set_read_fd(
+ ::_open_osfhandle(reinterpret_cast<intptr_t>(read_handle), O_RDONLY));
write_handle_.Reset(write_handle);
event_handle_.Reset(::CreateEvent(
&handles_are_inheritable,
@@ -756,27 +756,26 @@
nullptr)); // The even is unnamed.
GTEST_DEATH_TEST_CHECK_(event_handle_.Get() != nullptr);
const std::string filter_flag = std::string("--") + GTEST_FLAG_PREFIX_ +
- kFilterFlag + "=" + info->test_suite_name() +
- "." + info->name();
+ "filter=" + info->test_suite_name() + "." +
+ info->name();
const std::string internal_flag =
- std::string("--") + GTEST_FLAG_PREFIX_ + kInternalRunDeathTestFlag +
- "=" + file_ + "|" + StreamableToString(line_) + "|" +
- StreamableToString(death_test_index) + "|" +
+ std::string("--") + GTEST_FLAG_PREFIX_ +
+ "internal_run_death_test=" + file_ + "|" + StreamableToString(line_) +
+ "|" + StreamableToString(death_test_index) + "|" +
StreamableToString(static_cast<unsigned int>(::GetCurrentProcessId())) +
// size_t has the same width as pointers on both 32-bit and 64-bit
// Windows platforms.
// See http://msdn.microsoft.com/en-us/library/tcxf1dw6.aspx.
- "|" + StreamableToString(reinterpret_cast<size_t>(write_handle)) +
- "|" + StreamableToString(reinterpret_cast<size_t>(event_handle_.Get()));
+ "|" + StreamableToString(reinterpret_cast<size_t>(write_handle)) + "|" +
+ StreamableToString(reinterpret_cast<size_t>(event_handle_.Get()));
char executable_path[_MAX_PATH + 1]; // NOLINT
GTEST_DEATH_TEST_CHECK_(_MAX_PATH + 1 != ::GetModuleFileNameA(nullptr,
executable_path,
_MAX_PATH));
- std::string command_line =
- std::string(::GetCommandLineA()) + " " + filter_flag + " \"" +
- internal_flag + "\"";
+ std::string command_line = std::string(::GetCommandLineA()) + " " +
+ filter_flag + " \"" + internal_flag + "\"";
DeathTest::set_last_death_test_message("");
@@ -796,8 +795,8 @@
GTEST_DEATH_TEST_CHECK_(
::CreateProcessA(
executable_path, const_cast<char*>(command_line.c_str()),
- nullptr, // Retuned process handle is not inheritable.
- nullptr, // Retuned thread handle is not inheritable.
+ nullptr, // Returned process handle is not inheritable.
+ nullptr, // Returned thread handle is not inheritable.
TRUE, // Child inherits all inheritable handles (for write_handle_).
0x0, // Default creation flags.
nullptr, // Inherit the parent's environment.
@@ -809,7 +808,7 @@
return OVERSEE_TEST;
}
-# elif GTEST_OS_FUCHSIA
+#elif GTEST_OS_FUCHSIA
class FuchsiaDeathTest : public DeathTestImpl {
public:
@@ -855,18 +854,13 @@
template <typename Str>
void AddArguments(const ::std::vector<Str>& arguments) {
for (typename ::std::vector<Str>::const_iterator i = arguments.begin();
- i != arguments.end();
- ++i) {
+ i != arguments.end(); ++i) {
args_.insert(args_.end() - 1, posix::StrDup(i->c_str()));
}
}
- char* const* Argv() {
- return &args_[0];
- }
+ char* const* Argv() { return &args_[0]; }
- int size() {
- return static_cast<int>(args_.size()) - 1;
- }
+ int size() { return static_cast<int>(args_.size()) - 1; }
private:
std::vector<char*> args_;
@@ -880,8 +874,7 @@
const int kSocketKey = 1;
const int kExceptionKey = 2;
- if (!spawned())
- return 0;
+ if (!spawned()) return 0;
// Create a port to wait for socket/task/exception events.
zx_status_t status_zx;
@@ -890,8 +883,8 @@
GTEST_DEATH_TEST_CHECK_(status_zx == ZX_OK);
// Register to wait for the child process to terminate.
- status_zx = child_process_.wait_async(
- port, kProcessKey, ZX_PROCESS_TERMINATED, 0);
+ status_zx =
+ child_process_.wait_async(port, kProcessKey, ZX_PROCESS_TERMINATED, 0);
GTEST_DEATH_TEST_CHECK_(status_zx == ZX_OK);
// Register to wait for the socket to be readable or closed.
@@ -900,8 +893,8 @@
GTEST_DEATH_TEST_CHECK_(status_zx == ZX_OK);
// Register to wait for an exception.
- status_zx = exception_channel_.wait_async(
- port, kExceptionKey, ZX_CHANNEL_READABLE, 0);
+ status_zx = exception_channel_.wait_async(port, kExceptionKey,
+ ZX_CHANNEL_READABLE, 0);
GTEST_DEATH_TEST_CHECK_(status_zx == ZX_OK);
bool process_terminated = false;
@@ -931,9 +924,9 @@
size_t old_length = captured_stderr_.length();
size_t bytes_read = 0;
captured_stderr_.resize(old_length + kBufferSize);
- status_zx = stderr_socket_.read(
- 0, &captured_stderr_.front() + old_length, kBufferSize,
- &bytes_read);
+ status_zx =
+ stderr_socket_.read(0, &captured_stderr_.front() + old_length,
+ kBufferSize, &bytes_read);
captured_stderr_.resize(old_length + bytes_read);
} while (status_zx == ZX_OK);
if (status_zx == ZX_ERR_PEER_CLOSED) {
@@ -987,13 +980,12 @@
// Build the child process command line.
const std::string filter_flag = std::string("--") + GTEST_FLAG_PREFIX_ +
- kFilterFlag + "=" + info->test_suite_name() +
- "." + info->name();
- const std::string internal_flag =
- std::string("--") + GTEST_FLAG_PREFIX_ + kInternalRunDeathTestFlag + "="
- + file_ + "|"
- + StreamableToString(line_) + "|"
- + StreamableToString(death_test_index);
+ "filter=" + info->test_suite_name() + "." +
+ info->name();
+ const std::string internal_flag = std::string("--") + GTEST_FLAG_PREFIX_ +
+ kInternalRunDeathTestFlag + "=" + file_ +
+ "|" + StreamableToString(line_) + "|" +
+ StreamableToString(death_test_index);
Arguments args;
args.AddArguments(GetInjectableArgvs());
args.AddArgument(filter_flag.c_str());
@@ -1016,8 +1008,7 @@
// Create a socket pair will be used to receive the child process' stderr.
zx::socket stderr_producer_socket;
- status =
- zx::socket::create(0, &stderr_producer_socket, &stderr_socket_);
+ status = zx::socket::create(0, &stderr_producer_socket, &stderr_socket_);
GTEST_DEATH_TEST_CHECK_(status >= 0);
int stderr_producer_fd = -1;
status =
@@ -1034,35 +1025,32 @@
// Create a child job.
zx_handle_t child_job = ZX_HANDLE_INVALID;
- status = zx_job_create(zx_job_default(), 0, & child_job);
+ status = zx_job_create(zx_job_default(), 0, &child_job);
GTEST_DEATH_TEST_CHECK_(status == ZX_OK);
zx_policy_basic_t policy;
policy.condition = ZX_POL_NEW_ANY;
policy.policy = ZX_POL_ACTION_ALLOW;
- status = zx_job_set_policy(
- child_job, ZX_JOB_POL_RELATIVE, ZX_JOB_POL_BASIC, &policy, 1);
+ status = zx_job_set_policy(child_job, ZX_JOB_POL_RELATIVE, ZX_JOB_POL_BASIC,
+ &policy, 1);
GTEST_DEATH_TEST_CHECK_(status == ZX_OK);
// Create an exception channel attached to the |child_job|, to allow
// us to suppress the system default exception handler from firing.
- status =
- zx_task_create_exception_channel(
- child_job, 0, exception_channel_.reset_and_get_address());
+ status = zx_task_create_exception_channel(
+ child_job, 0, exception_channel_.reset_and_get_address());
GTEST_DEATH_TEST_CHECK_(status == ZX_OK);
// Spawn the child process.
- status = fdio_spawn_etc(
- child_job, FDIO_SPAWN_CLONE_ALL, args.Argv()[0], args.Argv(), nullptr,
- 2, spawn_actions, child_process_.reset_and_get_address(), nullptr);
+ status = fdio_spawn_etc(child_job, FDIO_SPAWN_CLONE_ALL, args.Argv()[0],
+ args.Argv(), nullptr, 2, spawn_actions,
+ child_process_.reset_and_get_address(), nullptr);
GTEST_DEATH_TEST_CHECK_(status == ZX_OK);
set_spawned(true);
return OVERSEE_TEST;
}
-std::string FuchsiaDeathTest::GetErrorLogs() {
- return captured_stderr_;
-}
+std::string FuchsiaDeathTest::GetErrorLogs() { return captured_stderr_; }
#else // We are neither on Windows, nor on Fuchsia.
@@ -1093,8 +1081,7 @@
// status, or 0 if no child process exists. As a side effect, sets the
// outcome data member.
int ForkingDeathTest::Wait() {
- if (!spawned())
- return 0;
+ if (!spawned()) return 0;
ReadAndInterpretStatusByte();
@@ -1173,11 +1160,11 @@
private:
static ::std::vector<std::string> GetArgvsForDeathTestChildProcess() {
::std::vector<std::string> args = GetInjectableArgvs();
-# if defined(GTEST_EXTRA_DEATH_TEST_COMMAND_LINE_ARGS_)
+#if defined(GTEST_EXTRA_DEATH_TEST_COMMAND_LINE_ARGS_)
::std::vector<std::string> extra_args =
GTEST_EXTRA_DEATH_TEST_COMMAND_LINE_ARGS_();
args.insert(args.end(), extra_args.begin(), extra_args.end());
-# endif // defined(GTEST_EXTRA_DEATH_TEST_COMMAND_LINE_ARGS_)
+#endif // defined(GTEST_EXTRA_DEATH_TEST_COMMAND_LINE_ARGS_)
return args;
}
// The name of the file in which the death test is located.
@@ -1204,14 +1191,11 @@
template <typename Str>
void AddArguments(const ::std::vector<Str>& arguments) {
for (typename ::std::vector<Str>::const_iterator i = arguments.begin();
- i != arguments.end();
- ++i) {
+ i != arguments.end(); ++i) {
args_.insert(args_.end() - 1, posix::StrDup(i->c_str()));
}
}
- char* const* Argv() {
- return &args_[0];
- }
+ char* const* Argv() { return &args_[0]; }
private:
std::vector<char*> args_;
@@ -1224,9 +1208,9 @@
int close_fd; // File descriptor to close; the read end of a pipe
};
-# if GTEST_OS_QNX
+#if GTEST_OS_QNX
extern "C" char** environ;
-# else // GTEST_OS_QNX
+#else // GTEST_OS_QNX
// The main function for a threadsafe-style death test child process.
// This function is called in a clone()-ed process and thus must avoid
// any potentially unsafe operations like malloc or libc functions.
@@ -1241,8 +1225,8 @@
UnitTest::GetInstance()->original_working_dir();
// We can safely call chdir() as it's a direct system call.
if (chdir(original_dir) != 0) {
- DeathTestAbort(std::string("chdir(\"") + original_dir + "\") failed: " +
- GetLastErrnoDescription());
+ DeathTestAbort(std::string("chdir(\"") + original_dir +
+ "\") failed: " + GetLastErrnoDescription());
return EXIT_FAILURE;
}
@@ -1253,13 +1237,12 @@
// one path separator.
execv(args->argv[0], args->argv);
DeathTestAbort(std::string("execv(") + args->argv[0] + ", ...) in " +
- original_dir + " failed: " +
- GetLastErrnoDescription());
+ original_dir + " failed: " + GetLastErrnoDescription());
return EXIT_FAILURE;
}
-# endif // GTEST_OS_QNX
+#endif // GTEST_OS_QNX
-# if GTEST_HAS_CLONE
+#if GTEST_HAS_CLONE
// Two utility routines that together determine the direction the stack
// grows.
// This could be accomplished more elegantly by a single recursive
@@ -1293,7 +1276,7 @@
StackLowerThanAddress(&dummy, &result);
return result;
}
-# endif // GTEST_HAS_CLONE
+#endif // GTEST_HAS_CLONE
// Spawns a child process with the same executable as the current process in
// a thread-safe manner and instructs it to run the death test. The
@@ -1303,10 +1286,10 @@
// spawn(2) there instead. The function dies with an error message if
// anything goes wrong.
static pid_t ExecDeathTestSpawnChild(char* const* argv, int close_fd) {
- ExecDeathTestArgs args = { argv, close_fd };
+ ExecDeathTestArgs args = {argv, close_fd};
pid_t child_pid = -1;
-# if GTEST_OS_QNX
+#if GTEST_OS_QNX
// Obtains the current directory and sets it to be closed in the child
// process.
const int cwd_fd = open(".", O_RDONLY);
@@ -1319,16 +1302,16 @@
UnitTest::GetInstance()->original_working_dir();
// We can safely call chdir() as it's a direct system call.
if (chdir(original_dir) != 0) {
- DeathTestAbort(std::string("chdir(\"") + original_dir + "\") failed: " +
- GetLastErrnoDescription());
+ DeathTestAbort(std::string("chdir(\"") + original_dir +
+ "\") failed: " + GetLastErrnoDescription());
return EXIT_FAILURE;
}
int fd_flags;
// Set close_fd to be closed after spawn.
GTEST_DEATH_TEST_CHECK_SYSCALL_(fd_flags = fcntl(close_fd, F_GETFD));
- GTEST_DEATH_TEST_CHECK_SYSCALL_(fcntl(close_fd, F_SETFD,
- fd_flags | FD_CLOEXEC));
+ GTEST_DEATH_TEST_CHECK_SYSCALL_(
+ fcntl(close_fd, F_SETFD, fd_flags | FD_CLOEXEC));
struct inheritance inherit = {0};
// spawn is a system call.
child_pid = spawn(args.argv[0], 0, nullptr, &inherit, args.argv, environ);
@@ -1336,8 +1319,8 @@
GTEST_DEATH_TEST_CHECK_(fchdir(cwd_fd) != -1);
GTEST_DEATH_TEST_CHECK_SYSCALL_(close(cwd_fd));
-# else // GTEST_OS_QNX
-# if GTEST_OS_LINUX
+#else // GTEST_OS_QNX
+#if GTEST_OS_LINUX
// When a SIGPROF signal is received while fork() or clone() are executing,
// the process may hang. To avoid this, we ignore SIGPROF here and re-enable
// it after the call to fork()/clone() is complete.
@@ -1346,12 +1329,12 @@
memset(&ignore_sigprof_action, 0, sizeof(ignore_sigprof_action));
sigemptyset(&ignore_sigprof_action.sa_mask);
ignore_sigprof_action.sa_handler = SIG_IGN;
- GTEST_DEATH_TEST_CHECK_SYSCALL_(sigaction(
- SIGPROF, &ignore_sigprof_action, &saved_sigprof_action));
-# endif // GTEST_OS_LINUX
+ GTEST_DEATH_TEST_CHECK_SYSCALL_(
+ sigaction(SIGPROF, &ignore_sigprof_action, &saved_sigprof_action));
+#endif // GTEST_OS_LINUX
-# if GTEST_HAS_CLONE
- const bool use_fork = GTEST_FLAG(death_test_use_fork);
+#if GTEST_HAS_CLONE
+ const bool use_fork = GTEST_FLAG_GET(death_test_use_fork);
if (!use_fork) {
static const bool stack_grows_down = StackGrowsDown();
@@ -1370,7 +1353,7 @@
const size_t kMaxStackAlignment = 64;
void* const stack_top =
static_cast<char*>(stack) +
- (stack_grows_down ? stack_size - kMaxStackAlignment : 0);
+ (stack_grows_down ? stack_size - kMaxStackAlignment : 0);
GTEST_DEATH_TEST_CHECK_(
static_cast<size_t>(stack_size) > kMaxStackAlignment &&
reinterpret_cast<uintptr_t>(stack_top) % kMaxStackAlignment == 0);
@@ -1379,19 +1362,19 @@
GTEST_DEATH_TEST_CHECK_(munmap(stack, stack_size) != -1);
}
-# else
+#else
const bool use_fork = true;
-# endif // GTEST_HAS_CLONE
+#endif // GTEST_HAS_CLONE
if (use_fork && (child_pid = fork()) == 0) {
- ExecDeathTestChildMain(&args);
- _exit(0);
+ ExecDeathTestChildMain(&args);
+ _exit(0);
}
-# endif // GTEST_OS_QNX
-# if GTEST_OS_LINUX
+#endif // GTEST_OS_QNX
+#if GTEST_OS_LINUX
GTEST_DEATH_TEST_CHECK_SYSCALL_(
sigaction(SIGPROF, &saved_sigprof_action, nullptr));
-# endif // GTEST_OS_LINUX
+#endif // GTEST_OS_LINUX
GTEST_DEATH_TEST_CHECK_(child_pid != -1);
return child_pid;
@@ -1420,13 +1403,13 @@
GTEST_DEATH_TEST_CHECK_(fcntl(pipe_fd[1], F_SETFD, 0) != -1);
const std::string filter_flag = std::string("--") + GTEST_FLAG_PREFIX_ +
- kFilterFlag + "=" + info->test_suite_name() +
- "." + info->name();
- const std::string internal_flag =
- std::string("--") + GTEST_FLAG_PREFIX_ + kInternalRunDeathTestFlag + "="
- + file_ + "|" + StreamableToString(line_) + "|"
- + StreamableToString(death_test_index) + "|"
- + StreamableToString(pipe_fd[1]);
+ "filter=" + info->test_suite_name() + "." +
+ info->name();
+ const std::string internal_flag = std::string("--") + GTEST_FLAG_PREFIX_ +
+ "internal_run_death_test=" + file_ + "|" +
+ StreamableToString(line_) + "|" +
+ StreamableToString(death_test_index) + "|" +
+ StreamableToString(pipe_fd[1]);
Arguments args;
args.AddArguments(GetArgvsForDeathTestChildProcess());
args.AddArgument(filter_flag.c_str());
@@ -1447,7 +1430,7 @@
return OVERSEE_TEST;
}
-# endif // !GTEST_OS_WINDOWS
+#endif // !GTEST_OS_WINDOWS
// Creates a concrete DeathTest-derived class that depends on the
// --gtest_death_test_style flag, and sets the pointer pointed to
@@ -1461,15 +1444,15 @@
UnitTestImpl* const impl = GetUnitTestImpl();
const InternalRunDeathTestFlag* const flag =
impl->internal_run_death_test_flag();
- const int death_test_index = impl->current_test_info()
- ->increment_death_test_count();
+ const int death_test_index =
+ impl->current_test_info()->increment_death_test_count();
if (flag != nullptr) {
if (death_test_index > flag->index()) {
DeathTest::set_last_death_test_message(
- "Death test count (" + StreamableToString(death_test_index)
- + ") somehow exceeded expected maximum ("
- + StreamableToString(flag->index()) + ")");
+ "Death test count (" + StreamableToString(death_test_index) +
+ ") somehow exceeded expected maximum (" +
+ StreamableToString(flag->index()) + ")");
return false;
}
@@ -1480,50 +1463,50 @@
}
}
-# if GTEST_OS_WINDOWS
+#if GTEST_OS_WINDOWS
- if (GTEST_FLAG(death_test_style) == "threadsafe" ||
- GTEST_FLAG(death_test_style) == "fast") {
+ if (GTEST_FLAG_GET(death_test_style) == "threadsafe" ||
+ GTEST_FLAG_GET(death_test_style) == "fast") {
*test = new WindowsDeathTest(statement, std::move(matcher), file, line);
}
-# elif GTEST_OS_FUCHSIA
+#elif GTEST_OS_FUCHSIA
- if (GTEST_FLAG(death_test_style) == "threadsafe" ||
- GTEST_FLAG(death_test_style) == "fast") {
+ if (GTEST_FLAG_GET(death_test_style) == "threadsafe" ||
+ GTEST_FLAG_GET(death_test_style) == "fast") {
*test = new FuchsiaDeathTest(statement, std::move(matcher), file, line);
}
-# else
+#else
- if (GTEST_FLAG(death_test_style) == "threadsafe") {
+ if (GTEST_FLAG_GET(death_test_style) == "threadsafe") {
*test = new ExecDeathTest(statement, std::move(matcher), file, line);
- } else if (GTEST_FLAG(death_test_style) == "fast") {
+ } else if (GTEST_FLAG_GET(death_test_style) == "fast") {
*test = new NoExecDeathTest(statement, std::move(matcher));
}
-# endif // GTEST_OS_WINDOWS
+#endif // GTEST_OS_WINDOWS
else { // NOLINT - this is more readable than unbalanced brackets inside #if.
- DeathTest::set_last_death_test_message(
- "Unknown death test style \"" + GTEST_FLAG(death_test_style)
- + "\" encountered");
+ DeathTest::set_last_death_test_message("Unknown death test style \"" +
+ GTEST_FLAG_GET(death_test_style) +
+ "\" encountered");
return false;
}
return true;
}
-# if GTEST_OS_WINDOWS
+#if GTEST_OS_WINDOWS
// Recreates the pipe and event handles from the provided parameters,
// signals the event, and returns a file descriptor wrapped around the pipe
// handle. This function is called in the child process only.
static int GetStatusFileDescriptor(unsigned int parent_process_id,
- size_t write_handle_as_size_t,
- size_t event_handle_as_size_t) {
+ size_t write_handle_as_size_t,
+ size_t event_handle_as_size_t) {
AutoHandle parent_process_handle(::OpenProcess(PROCESS_DUP_HANDLE,
- FALSE, // Non-inheritable.
- parent_process_id));
+ FALSE, // Non-inheritable.
+ parent_process_id));
if (parent_process_handle.Get() == INVALID_HANDLE_VALUE) {
DeathTestAbort("Unable to open parent process " +
StreamableToString(parent_process_id));
@@ -1531,8 +1514,7 @@
GTEST_CHECK_(sizeof(HANDLE) <= sizeof(size_t));
- const HANDLE write_handle =
- reinterpret_cast<HANDLE>(write_handle_as_size_t);
+ const HANDLE write_handle = reinterpret_cast<HANDLE>(write_handle_as_size_t);
HANDLE dup_write_handle;
// The newly initialized handle is accessible only in the parent
@@ -1554,9 +1536,7 @@
HANDLE dup_event_handle;
if (!::DuplicateHandle(parent_process_handle.Get(), event_handle,
- ::GetCurrentProcess(), &dup_event_handle,
- 0x0,
- FALSE,
+ ::GetCurrentProcess(), &dup_event_handle, 0x0, FALSE,
DUPLICATE_SAME_ACCESS)) {
DeathTestAbort("Unable to duplicate the event handle " +
StreamableToString(event_handle_as_size_t) +
@@ -1578,61 +1558,57 @@
return write_fd;
}
-# endif // GTEST_OS_WINDOWS
+#endif // GTEST_OS_WINDOWS
// Returns a newly created InternalRunDeathTestFlag object with fields
// initialized from the GTEST_FLAG(internal_run_death_test) flag if
// the flag is specified; otherwise returns NULL.
InternalRunDeathTestFlag* ParseInternalRunDeathTestFlag() {
- if (GTEST_FLAG(internal_run_death_test) == "") return nullptr;
+ if (GTEST_FLAG_GET(internal_run_death_test) == "") return nullptr;
// GTEST_HAS_DEATH_TEST implies that we have ::std::string, so we
// can use it here.
int line = -1;
int index = -1;
::std::vector< ::std::string> fields;
- SplitString(GTEST_FLAG(internal_run_death_test).c_str(), '|', &fields);
+ SplitString(GTEST_FLAG_GET(internal_run_death_test), '|', &fields);
int write_fd = -1;
-# if GTEST_OS_WINDOWS
+#if GTEST_OS_WINDOWS
unsigned int parent_process_id = 0;
size_t write_handle_as_size_t = 0;
size_t event_handle_as_size_t = 0;
- if (fields.size() != 6
- || !ParseNaturalNumber(fields[1], &line)
- || !ParseNaturalNumber(fields[2], &index)
- || !ParseNaturalNumber(fields[3], &parent_process_id)
- || !ParseNaturalNumber(fields[4], &write_handle_as_size_t)
- || !ParseNaturalNumber(fields[5], &event_handle_as_size_t)) {
+ if (fields.size() != 6 || !ParseNaturalNumber(fields[1], &line) ||
+ !ParseNaturalNumber(fields[2], &index) ||
+ !ParseNaturalNumber(fields[3], &parent_process_id) ||
+ !ParseNaturalNumber(fields[4], &write_handle_as_size_t) ||
+ !ParseNaturalNumber(fields[5], &event_handle_as_size_t)) {
DeathTestAbort("Bad --gtest_internal_run_death_test flag: " +
- GTEST_FLAG(internal_run_death_test));
+ GTEST_FLAG_GET(internal_run_death_test));
}
- write_fd = GetStatusFileDescriptor(parent_process_id,
- write_handle_as_size_t,
+ write_fd = GetStatusFileDescriptor(parent_process_id, write_handle_as_size_t,
event_handle_as_size_t);
-# elif GTEST_OS_FUCHSIA
+#elif GTEST_OS_FUCHSIA
- if (fields.size() != 3
- || !ParseNaturalNumber(fields[1], &line)
- || !ParseNaturalNumber(fields[2], &index)) {
- DeathTestAbort("Bad --gtest_internal_run_death_test flag: "
- + GTEST_FLAG(internal_run_death_test));
+ if (fields.size() != 3 || !ParseNaturalNumber(fields[1], &line) ||
+ !ParseNaturalNumber(fields[2], &index)) {
+ DeathTestAbort("Bad --gtest_internal_run_death_test flag: " +
+ GTEST_FLAG_GET(internal_run_death_test));
}
-# else
+#else
- if (fields.size() != 4
- || !ParseNaturalNumber(fields[1], &line)
- || !ParseNaturalNumber(fields[2], &index)
- || !ParseNaturalNumber(fields[3], &write_fd)) {
- DeathTestAbort("Bad --gtest_internal_run_death_test flag: "
- + GTEST_FLAG(internal_run_death_test));
+ if (fields.size() != 4 || !ParseNaturalNumber(fields[1], &line) ||
+ !ParseNaturalNumber(fields[2], &index) ||
+ !ParseNaturalNumber(fields[3], &write_fd)) {
+ DeathTestAbort("Bad --gtest_internal_run_death_test flag: " +
+ GTEST_FLAG_GET(internal_run_death_test));
}
-# endif // GTEST_OS_WINDOWS
+#endif // GTEST_OS_WINDOWS
return new InternalRunDeathTestFlag(fields[0], line, index, write_fd);
}
diff --git a/third_party/googletest/src/googletest/src/gtest-filepath.cc b/third_party/googletest/src/googletest/src/gtest-filepath.cc
index 0b56294..f6ee90c 100644
--- a/third_party/googletest/src/googletest/src/gtest-filepath.cc
+++ b/third_party/googletest/src/googletest/src/gtest-filepath.cc
@@ -30,29 +30,31 @@
#include "gtest/internal/gtest-filepath.h"
#include <stdlib.h>
-#include "gtest/internal/gtest-port.h"
+
#include "gtest/gtest-message.h"
+#include "gtest/internal/gtest-port.h"
#if GTEST_OS_WINDOWS_MOBILE
-# include <windows.h>
+#include <windows.h>
#elif GTEST_OS_WINDOWS
-# include <direct.h>
-# include <io.h>
+#include <direct.h>
+#include <io.h>
#else
-# include <limits.h>
-# include <climits> // Some Linux distributions define PATH_MAX here.
-#endif // GTEST_OS_WINDOWS_MOBILE
+#include <limits.h>
+
+#include <climits> // Some Linux distributions define PATH_MAX here.
+#endif // GTEST_OS_WINDOWS_MOBILE
#include "gtest/internal/gtest-string.h"
#if GTEST_OS_WINDOWS
-# define GTEST_PATH_MAX_ _MAX_PATH
+#define GTEST_PATH_MAX_ _MAX_PATH
#elif defined(PATH_MAX)
-# define GTEST_PATH_MAX_ PATH_MAX
+#define GTEST_PATH_MAX_ PATH_MAX
#elif defined(_XOPEN_PATH_MAX)
-# define GTEST_PATH_MAX_ _XOPEN_PATH_MAX
+#define GTEST_PATH_MAX_ _XOPEN_PATH_MAX
#else
-# define GTEST_PATH_MAX_ _POSIX_PATH_MAX
+#define GTEST_PATH_MAX_ _POSIX_PATH_MAX
#endif // GTEST_OS_WINDOWS
namespace testing {
@@ -66,16 +68,16 @@
const char kPathSeparator = '\\';
const char kAlternatePathSeparator = '/';
const char kAlternatePathSeparatorString[] = "/";
-# if GTEST_OS_WINDOWS_MOBILE
+#if GTEST_OS_WINDOWS_MOBILE
// Windows CE doesn't have a current directory. You should not use
// the current directory in tests on Windows CE, but this at least
// provides a reasonable fallback.
const char kCurrentDirectoryString[] = "\\";
// Windows CE doesn't define INVALID_FILE_ATTRIBUTES
const DWORD kInvalidFileAttributes = 0xffffffff;
-# else
+#else
const char kCurrentDirectoryString[] = ".\\";
-# endif // GTEST_OS_WINDOWS_MOBILE
+#endif // GTEST_OS_WINDOWS_MOBILE
#else
const char kPathSeparator = '/';
const char kCurrentDirectoryString[] = "./";
@@ -99,17 +101,17 @@
// something reasonable.
return FilePath(kCurrentDirectoryString);
#elif GTEST_OS_WINDOWS
- char cwd[GTEST_PATH_MAX_ + 1] = { '\0' };
+ char cwd[GTEST_PATH_MAX_ + 1] = {'\0'};
return FilePath(_getcwd(cwd, sizeof(cwd)) == nullptr ? "" : cwd);
#else
- char cwd[GTEST_PATH_MAX_ + 1] = { '\0' };
+ char cwd[GTEST_PATH_MAX_ + 1] = {'\0'};
char* result = getcwd(cwd, sizeof(cwd));
-# if GTEST_OS_NACL
+#if GTEST_OS_NACL
// getcwd will likely fail in NaCl due to the sandbox, so return something
// reasonable. The user may have provided a shim implementation for getcwd,
// however, so fallback only when failure is detected.
return FilePath(result == nullptr ? kCurrentDirectoryString : cwd);
-# endif // GTEST_OS_NACL
+#endif // GTEST_OS_NACL
return FilePath(result == nullptr ? "" : cwd);
#endif // GTEST_OS_WINDOWS_MOBILE
}
@@ -121,8 +123,8 @@
FilePath FilePath::RemoveExtension(const char* extension) const {
const std::string dot_extension = std::string(".") + extension;
if (String::EndsWithCaseInsensitive(pathname_, dot_extension)) {
- return FilePath(pathname_.substr(
- 0, pathname_.length() - dot_extension.length()));
+ return FilePath(
+ pathname_.substr(0, pathname_.length() - dot_extension.length()));
}
return *this;
}
@@ -178,15 +180,14 @@
// than zero (e.g., 12), returns "dir/test_12.xml".
// On Windows platform, uses \ as the separator rather than /.
FilePath FilePath::MakeFileName(const FilePath& directory,
- const FilePath& base_name,
- int number,
+ const FilePath& base_name, int number,
const char* extension) {
std::string file;
if (number == 0) {
file = base_name.string() + "." + extension;
} else {
- file = base_name.string() + "_" + StreamableToString(number)
- + "." + extension;
+ file =
+ base_name.string() + "_" + StreamableToString(number) + "." + extension;
}
return ConcatPaths(directory, FilePath(file));
}
@@ -195,8 +196,7 @@
// On Windows, uses \ as the separator rather than /.
FilePath FilePath::ConcatPaths(const FilePath& directory,
const FilePath& relative_path) {
- if (directory.IsEmpty())
- return relative_path;
+ if (directory.IsEmpty()) return relative_path;
const FilePath dir(directory.RemoveTrailingPathSeparator());
return FilePath(dir.string() + kPathSeparator + relative_path.string());
}
@@ -207,7 +207,7 @@
#if GTEST_OS_WINDOWS_MOBILE
LPCWSTR unicode = String::AnsiToUtf16(pathname_.c_str());
const DWORD attributes = GetFileAttributes(unicode);
- delete [] unicode;
+ delete[] unicode;
return attributes != kInvalidFileAttributes;
#else
posix::StatStruct file_stat{};
@@ -222,8 +222,8 @@
#if GTEST_OS_WINDOWS
// Don't strip off trailing separator if path is a root directory on
// Windows (like "C:\\").
- const FilePath& path(IsRootDirectory() ? *this :
- RemoveTrailingPathSeparator());
+ const FilePath& path(IsRootDirectory() ? *this
+ : RemoveTrailingPathSeparator());
#else
const FilePath& path(*this);
#endif
@@ -231,15 +231,15 @@
#if GTEST_OS_WINDOWS_MOBILE
LPCWSTR unicode = String::AnsiToUtf16(path.c_str());
const DWORD attributes = GetFileAttributes(unicode);
- delete [] unicode;
+ delete[] unicode;
if ((attributes != kInvalidFileAttributes) &&
(attributes & FILE_ATTRIBUTE_DIRECTORY)) {
result = true;
}
#else
posix::StatStruct file_stat{};
- result = posix::Stat(path.c_str(), &file_stat) == 0 &&
- posix::IsDir(file_stat);
+ result =
+ posix::Stat(path.c_str(), &file_stat) == 0 && posix::IsDir(file_stat);
#endif // GTEST_OS_WINDOWS_MOBILE
return result;
@@ -260,10 +260,9 @@
const char* const name = pathname_.c_str();
#if GTEST_OS_WINDOWS
return pathname_.length() >= 3 &&
- ((name[0] >= 'a' && name[0] <= 'z') ||
- (name[0] >= 'A' && name[0] <= 'Z')) &&
- name[1] == ':' &&
- IsPathSeparator(name[2]);
+ ((name[0] >= 'a' && name[0] <= 'z') ||
+ (name[0] >= 'A' && name[0] <= 'Z')) &&
+ name[1] == ':' && IsPathSeparator(name[2]);
#else
return IsPathSeparator(name[0]);
#endif
@@ -321,7 +320,7 @@
FilePath removed_sep(this->RemoveTrailingPathSeparator());
LPCWSTR unicode = String::AnsiToUtf16(removed_sep.c_str());
int result = CreateDirectory(unicode, nullptr) ? 0 : -1;
- delete [] unicode;
+ delete[] unicode;
#elif GTEST_OS_WINDOWS
int result = _mkdir(pathname_.c_str());
#elif GTEST_OS_ESP8266 || GTEST_OS_XTENSA
@@ -341,9 +340,8 @@
// name, otherwise return the name string unmodified.
// On Windows platform, uses \ as the separator, other platforms use /.
FilePath FilePath::RemoveTrailingPathSeparator() const {
- return IsDirectory()
- ? FilePath(pathname_.substr(0, pathname_.length() - 1))
- : *this;
+ return IsDirectory() ? FilePath(pathname_.substr(0, pathname_.length() - 1))
+ : *this;
}
// Removes any redundant separators that might be in the pathname.
diff --git a/third_party/googletest/src/googletest/src/gtest-internal-inl.h b/third_party/googletest/src/googletest/src/gtest-internal-inl.h
index 6d8cecb..0b9e929 100644
--- a/third_party/googletest/src/googletest/src/gtest-internal-inl.h
+++ b/third_party/googletest/src/googletest/src/gtest-internal-inl.h
@@ -35,7 +35,7 @@
#define GOOGLETEST_SRC_GTEST_INTERNAL_INL_H_
#ifndef _WIN32_WCE
-# include <errno.h>
+#include <errno.h>
#endif // !_WIN32_WCE
#include <stddef.h>
#include <stdlib.h> // For strtoll/_strtoul64/malloc/free.
@@ -50,22 +50,20 @@
#include "gtest/internal/gtest-port.h"
#if GTEST_CAN_STREAM_RESULTS_
-# include <arpa/inet.h> // NOLINT
-# include <netdb.h> // NOLINT
+#include <arpa/inet.h> // NOLINT
+#include <netdb.h> // NOLINT
#endif
#if GTEST_OS_WINDOWS
-# include <windows.h> // NOLINT
-#endif // GTEST_OS_WINDOWS
+#include <windows.h> // NOLINT
+#endif // GTEST_OS_WINDOWS
-#include "gtest/gtest.h"
#include "gtest/gtest-spi.h"
+#include "gtest/gtest.h"
GTEST_DISABLE_MSC_WARNINGS_PUSH_(4251 \
/* class A needs to have dll-interface to be used by clients of class B */)
-namespace testing {
-
// Declares the flags.
//
// We don't want the users to modify this flag in the code, but want
@@ -73,32 +71,13 @@
// declare it here as opposed to in gtest.h.
GTEST_DECLARE_bool_(death_test_use_fork);
+namespace testing {
namespace internal {
// The value of GetTestTypeId() as seen from within the Google Test
// library. This is solely for testing GetTestTypeId().
GTEST_API_ extern const TypeId kTestTypeIdInGoogleTest;
-// Names of the flags (needed for parsing Google Test flags).
-const char kAlsoRunDisabledTestsFlag[] = "also_run_disabled_tests";
-const char kBreakOnFailureFlag[] = "break_on_failure";
-const char kCatchExceptionsFlag[] = "catch_exceptions";
-const char kColorFlag[] = "color";
-const char kFailFast[] = "fail_fast";
-const char kFilterFlag[] = "filter";
-const char kListTestsFlag[] = "list_tests";
-const char kOutputFlag[] = "output";
-const char kBriefFlag[] = "brief";
-const char kPrintTimeFlag[] = "print_time";
-const char kPrintUTF8Flag[] = "print_utf8";
-const char kRandomSeedFlag[] = "random_seed";
-const char kRepeatFlag[] = "repeat";
-const char kShuffleFlag[] = "shuffle";
-const char kStackTraceDepthFlag[] = "stack_trace_depth";
-const char kStreamResultToFlag[] = "stream_result_to";
-const char kThrowOnFailureFlag[] = "throw_on_failure";
-const char kFlagfileFlag[] = "flagfile";
-
// A valid random seed must be in [1, kMaxRandomSeed].
const int kMaxRandomSeed = 99999;
@@ -125,21 +104,21 @@
//
// On success, stores the value of the flag in *value, and returns
// true. On failure, returns false without changing *value.
-GTEST_API_ bool ParseInt32Flag(
- const char* str, const char* flag, int32_t* value);
+GTEST_API_ bool ParseFlag(const char* str, const char* flag, int32_t* value);
// Returns a random seed in range [1, kMaxRandomSeed] based on the
// given --gtest_random_seed flag value.
inline int GetRandomSeedFromFlag(int32_t random_seed_flag) {
- const unsigned int raw_seed = (random_seed_flag == 0) ?
- static_cast<unsigned int>(GetTimeInMillis()) :
- static_cast<unsigned int>(random_seed_flag);
+ const unsigned int raw_seed =
+ (random_seed_flag == 0) ? static_cast<unsigned int>(GetTimeInMillis())
+ : static_cast<unsigned int>(random_seed_flag);
// Normalizes the actual seed to range [1, kMaxRandomSeed] such that
// it's easy to type.
const int normalized_seed =
static_cast<int>((raw_seed - 1U) %
- static_cast<unsigned int>(kMaxRandomSeed)) + 1;
+ static_cast<unsigned int>(kMaxRandomSeed)) +
+ 1;
return normalized_seed;
}
@@ -160,50 +139,54 @@
public:
// The c'tor.
GTestFlagSaver() {
- also_run_disabled_tests_ = GTEST_FLAG(also_run_disabled_tests);
- break_on_failure_ = GTEST_FLAG(break_on_failure);
- catch_exceptions_ = GTEST_FLAG(catch_exceptions);
- color_ = GTEST_FLAG(color);
- death_test_style_ = GTEST_FLAG(death_test_style);
- death_test_use_fork_ = GTEST_FLAG(death_test_use_fork);
- fail_fast_ = GTEST_FLAG(fail_fast);
- filter_ = GTEST_FLAG(filter);
- internal_run_death_test_ = GTEST_FLAG(internal_run_death_test);
- list_tests_ = GTEST_FLAG(list_tests);
- output_ = GTEST_FLAG(output);
- brief_ = GTEST_FLAG(brief);
- print_time_ = GTEST_FLAG(print_time);
- print_utf8_ = GTEST_FLAG(print_utf8);
- random_seed_ = GTEST_FLAG(random_seed);
- repeat_ = GTEST_FLAG(repeat);
- shuffle_ = GTEST_FLAG(shuffle);
- stack_trace_depth_ = GTEST_FLAG(stack_trace_depth);
- stream_result_to_ = GTEST_FLAG(stream_result_to);
- throw_on_failure_ = GTEST_FLAG(throw_on_failure);
+ also_run_disabled_tests_ = GTEST_FLAG_GET(also_run_disabled_tests);
+ break_on_failure_ = GTEST_FLAG_GET(break_on_failure);
+ catch_exceptions_ = GTEST_FLAG_GET(catch_exceptions);
+ color_ = GTEST_FLAG_GET(color);
+ death_test_style_ = GTEST_FLAG_GET(death_test_style);
+ death_test_use_fork_ = GTEST_FLAG_GET(death_test_use_fork);
+ fail_fast_ = GTEST_FLAG_GET(fail_fast);
+ filter_ = GTEST_FLAG_GET(filter);
+ internal_run_death_test_ = GTEST_FLAG_GET(internal_run_death_test);
+ list_tests_ = GTEST_FLAG_GET(list_tests);
+ output_ = GTEST_FLAG_GET(output);
+ brief_ = GTEST_FLAG_GET(brief);
+ print_time_ = GTEST_FLAG_GET(print_time);
+ print_utf8_ = GTEST_FLAG_GET(print_utf8);
+ random_seed_ = GTEST_FLAG_GET(random_seed);
+ repeat_ = GTEST_FLAG_GET(repeat);
+ recreate_environments_when_repeating_ =
+ GTEST_FLAG_GET(recreate_environments_when_repeating);
+ shuffle_ = GTEST_FLAG_GET(shuffle);
+ stack_trace_depth_ = GTEST_FLAG_GET(stack_trace_depth);
+ stream_result_to_ = GTEST_FLAG_GET(stream_result_to);
+ throw_on_failure_ = GTEST_FLAG_GET(throw_on_failure);
}
// The d'tor is not virtual. DO NOT INHERIT FROM THIS CLASS.
~GTestFlagSaver() {
- GTEST_FLAG(also_run_disabled_tests) = also_run_disabled_tests_;
- GTEST_FLAG(break_on_failure) = break_on_failure_;
- GTEST_FLAG(catch_exceptions) = catch_exceptions_;
- GTEST_FLAG(color) = color_;
- GTEST_FLAG(death_test_style) = death_test_style_;
- GTEST_FLAG(death_test_use_fork) = death_test_use_fork_;
- GTEST_FLAG(filter) = filter_;
- GTEST_FLAG(fail_fast) = fail_fast_;
- GTEST_FLAG(internal_run_death_test) = internal_run_death_test_;
- GTEST_FLAG(list_tests) = list_tests_;
- GTEST_FLAG(output) = output_;
- GTEST_FLAG(brief) = brief_;
- GTEST_FLAG(print_time) = print_time_;
- GTEST_FLAG(print_utf8) = print_utf8_;
- GTEST_FLAG(random_seed) = random_seed_;
- GTEST_FLAG(repeat) = repeat_;
- GTEST_FLAG(shuffle) = shuffle_;
- GTEST_FLAG(stack_trace_depth) = stack_trace_depth_;
- GTEST_FLAG(stream_result_to) = stream_result_to_;
- GTEST_FLAG(throw_on_failure) = throw_on_failure_;
+ GTEST_FLAG_SET(also_run_disabled_tests, also_run_disabled_tests_);
+ GTEST_FLAG_SET(break_on_failure, break_on_failure_);
+ GTEST_FLAG_SET(catch_exceptions, catch_exceptions_);
+ GTEST_FLAG_SET(color, color_);
+ GTEST_FLAG_SET(death_test_style, death_test_style_);
+ GTEST_FLAG_SET(death_test_use_fork, death_test_use_fork_);
+ GTEST_FLAG_SET(filter, filter_);
+ GTEST_FLAG_SET(fail_fast, fail_fast_);
+ GTEST_FLAG_SET(internal_run_death_test, internal_run_death_test_);
+ GTEST_FLAG_SET(list_tests, list_tests_);
+ GTEST_FLAG_SET(output, output_);
+ GTEST_FLAG_SET(brief, brief_);
+ GTEST_FLAG_SET(print_time, print_time_);
+ GTEST_FLAG_SET(print_utf8, print_utf8_);
+ GTEST_FLAG_SET(random_seed, random_seed_);
+ GTEST_FLAG_SET(repeat, repeat_);
+ GTEST_FLAG_SET(recreate_environments_when_repeating,
+ recreate_environments_when_repeating_);
+ GTEST_FLAG_SET(shuffle, shuffle_);
+ GTEST_FLAG_SET(stack_trace_depth, stack_trace_depth_);
+ GTEST_FLAG_SET(stream_result_to, stream_result_to_);
+ GTEST_FLAG_SET(throw_on_failure, throw_on_failure_);
}
private:
@@ -224,6 +207,7 @@
bool print_utf8_;
int32_t random_seed_;
int32_t repeat_;
+ bool recreate_environments_when_repeating_;
bool shuffle_;
int32_t stack_trace_depth_;
std::string stream_result_to_;
@@ -278,8 +262,8 @@
// returns true if and only if the test should be run on this shard. The test id
// is some arbitrary but unique non-negative integer assigned to each test
// method. Assumes that 0 <= shard_index < total_shards.
-GTEST_API_ bool ShouldRunTestOnShard(
- int total_shards, int shard_index, int test_id);
+GTEST_API_ bool ShouldRunTestOnShard(int total_shards, int shard_index,
+ int test_id);
// STL container utilities.
@@ -290,9 +274,8 @@
// Implemented as an explicit loop since std::count_if() in libCstd on
// Solaris has a non-standard signature.
int count = 0;
- for (typename Container::const_iterator it = c.begin(); it != c.end(); ++it) {
- if (predicate(*it))
- ++count;
+ for (auto it = c.begin(); it != c.end(); ++it) {
+ if (predicate(*it)) ++count;
}
return count;
}
@@ -441,7 +424,9 @@
static const char* const kElidedFramesMarker;
private:
- GTEST_DISALLOW_COPY_AND_ASSIGN_(OsStackTraceGetterInterface);
+ OsStackTraceGetterInterface(const OsStackTraceGetterInterface&) = delete;
+ OsStackTraceGetterInterface& operator=(const OsStackTraceGetterInterface&) =
+ delete;
};
// A working implementation of the OsStackTraceGetterInterface interface.
@@ -463,7 +448,8 @@
void* caller_frame_ = nullptr;
#endif // GTEST_HAS_ABSL
- GTEST_DISALLOW_COPY_AND_ASSIGN_(OsStackTraceGetter);
+ OsStackTraceGetter(const OsStackTraceGetter&) = delete;
+ OsStackTraceGetter& operator=(const OsStackTraceGetter&) = delete;
};
// Information about a Google Test trace point.
@@ -476,7 +462,7 @@
// This is the default global test part result reporter used in UnitTestImpl.
// This class should only be used by UnitTestImpl.
class DefaultGlobalTestPartResultReporter
- : public TestPartResultReporterInterface {
+ : public TestPartResultReporterInterface {
public:
explicit DefaultGlobalTestPartResultReporter(UnitTestImpl* unit_test);
// Implements the TestPartResultReporterInterface. Reports the test part
@@ -486,7 +472,10 @@
private:
UnitTestImpl* const unit_test_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(DefaultGlobalTestPartResultReporter);
+ DefaultGlobalTestPartResultReporter(
+ const DefaultGlobalTestPartResultReporter&) = delete;
+ DefaultGlobalTestPartResultReporter& operator=(
+ const DefaultGlobalTestPartResultReporter&) = delete;
};
// This is the default per thread test part result reporter used in
@@ -502,7 +491,10 @@
private:
UnitTestImpl* const unit_test_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(DefaultPerThreadTestPartResultReporter);
+ DefaultPerThreadTestPartResultReporter(
+ const DefaultPerThreadTestPartResultReporter&) = delete;
+ DefaultPerThreadTestPartResultReporter& operator=(
+ const DefaultPerThreadTestPartResultReporter&) = delete;
};
// The private implementation of the UnitTest class. We don't protect
@@ -640,7 +632,8 @@
// For example, if Foo() calls Bar(), which in turn calls
// CurrentOsStackTraceExceptTop(1), Foo() will be included in the
// trace but Bar() and CurrentOsStackTraceExceptTop() won't.
- std::string CurrentOsStackTraceExceptTop(int skip_count) GTEST_NO_INLINE_;
+ std::string CurrentOsStackTraceExceptTop(int skip_count)
+ GTEST_NO_INLINE_ GTEST_NO_TAIL_CALL_;
// Finds and returns a TestSuite with the given name. If one doesn't
// exist, creates one and returns it.
@@ -744,9 +737,7 @@
}
// Clears the results of ad-hoc test assertions.
- void ClearAdHocTestResult() {
- ad_hoc_test_result_.Clear();
- }
+ void ClearAdHocTestResult() { ad_hoc_test_result_.Clear(); }
// Adds a TestProperty to the current TestResult object when invoked in a
// context of a test or a test suite, or to the global property set. If the
@@ -754,10 +745,7 @@
// updated.
void RecordProperty(const TestProperty& test_property);
- enum ReactionToSharding {
- HONOR_SHARDING_PROTOCOL,
- IGNORE_SHARDING_PROTOCOL
- };
+ enum ReactionToSharding { HONOR_SHARDING_PROTOCOL, IGNORE_SHARDING_PROTOCOL };
// Matches the full name of each test against the user-specified
// filter to decide whether the test should run, then records the
@@ -963,7 +951,8 @@
// starts.
bool catch_exceptions_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(UnitTestImpl);
+ UnitTestImpl(const UnitTestImpl&) = delete;
+ UnitTestImpl& operator=(const UnitTestImpl&) = delete;
}; // class UnitTestImpl
// Convenience function for accessing the global UnitTest
@@ -986,8 +975,9 @@
GTEST_API_ bool AtomMatchesChar(bool escaped, char pattern, char ch);
GTEST_API_ bool ValidateRegex(const char* regex);
GTEST_API_ bool MatchRegexAtHead(const char* regex, const char* str);
-GTEST_API_ bool MatchRepetitionAndRegexAtHead(
- bool escaped, char ch, char repeat, const char* regex, const char* str);
+GTEST_API_ bool MatchRepetitionAndRegexAtHead(bool escaped, char ch,
+ char repeat, const char* regex,
+ const char* str);
GTEST_API_ bool MatchRegexAnywhere(const char* regex, const char* str);
#endif // GTEST_USES_SIMPLE_RE
@@ -1089,8 +1079,7 @@
}
~SocketWriter() override {
- if (sockfd_ != -1)
- CloseConnection();
+ if (sockfd_ != -1) CloseConnection();
}
// Sends a string to the socket.
@@ -1100,9 +1089,8 @@
const auto len = static_cast<size_t>(message.length());
if (write(sockfd_, message.c_str(), len) != static_cast<ssize_t>(len)) {
- GTEST_LOG_(WARNING)
- << "stream_result_to: failed to stream to "
- << host_name_ << ":" << port_num_;
+ GTEST_LOG_(WARNING) << "stream_result_to: failed to stream to "
+ << host_name_ << ":" << port_num_;
}
}
@@ -1123,7 +1111,8 @@
const std::string host_name_;
const std::string port_num_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(SocketWriter);
+ SocketWriter(const SocketWriter&) = delete;
+ SocketWriter& operator=(const SocketWriter&) = delete;
}; // class SocketWriter
// Escapes '=', '&', '%', and '\n' characters in str as "%xx".
@@ -1135,7 +1124,9 @@
}
explicit StreamingListener(AbstractSocketWriter* socket_writer)
- : socket_writer_(socket_writer) { Start(); }
+ : socket_writer_(socket_writer) {
+ Start();
+ }
void OnTestProgramStart(const UnitTest& /* unit_test */) override {
SendLn("event=TestProgramStart");
@@ -1158,22 +1149,22 @@
void OnTestIterationEnd(const UnitTest& unit_test,
int /* iteration */) override {
- SendLn("event=TestIterationEnd&passed=" +
- FormatBool(unit_test.Passed()) + "&elapsed_time=" +
- StreamableToString(unit_test.elapsed_time()) + "ms");
+ SendLn("event=TestIterationEnd&passed=" + FormatBool(unit_test.Passed()) +
+ "&elapsed_time=" + StreamableToString(unit_test.elapsed_time()) +
+ "ms");
}
// Note that "event=TestCaseStart" is a wire format and has to remain
// "case" for compatibility
- void OnTestCaseStart(const TestCase& test_case) override {
- SendLn(std::string("event=TestCaseStart&name=") + test_case.name());
+ void OnTestSuiteStart(const TestSuite& test_suite) override {
+ SendLn(std::string("event=TestCaseStart&name=") + test_suite.name());
}
// Note that "event=TestCaseEnd" is a wire format and has to remain
// "case" for compatibility
- void OnTestCaseEnd(const TestCase& test_case) override {
- SendLn("event=TestCaseEnd&passed=" + FormatBool(test_case.Passed()) +
- "&elapsed_time=" + StreamableToString(test_case.elapsed_time()) +
+ void OnTestSuiteEnd(const TestSuite& test_suite) override {
+ SendLn("event=TestCaseEnd&passed=" + FormatBool(test_suite.Passed()) +
+ "&elapsed_time=" + StreamableToString(test_suite.elapsed_time()) +
"ms");
}
@@ -1183,8 +1174,7 @@
void OnTestEnd(const TestInfo& test_info) override {
SendLn("event=TestEnd&passed=" +
- FormatBool((test_info.result())->Passed()) +
- "&elapsed_time=" +
+ FormatBool((test_info.result())->Passed()) + "&elapsed_time=" +
StreamableToString((test_info.result())->elapsed_time()) + "ms");
}
@@ -1208,7 +1198,8 @@
const std::unique_ptr<AbstractSocketWriter> socket_writer_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(StreamingListener);
+ StreamingListener(const StreamingListener&) = delete;
+ StreamingListener& operator=(const StreamingListener&) = delete;
}; // class StreamingListener
#endif // GTEST_CAN_STREAM_RESULTS_
diff --git a/third_party/googletest/src/googletest/src/gtest-matchers.cc b/third_party/googletest/src/googletest/src/gtest-matchers.cc
index 65104eb..7e3bcc0 100644
--- a/third_party/googletest/src/googletest/src/gtest-matchers.cc
+++ b/third_party/googletest/src/googletest/src/gtest-matchers.cc
@@ -32,12 +32,13 @@
// This file implements just enough of the matcher interface to allow
// EXPECT_DEATH and friends to accept a matcher argument.
-#include "gtest/internal/gtest-internal.h"
-#include "gtest/internal/gtest-port.h"
#include "gtest/gtest-matchers.h"
#include <string>
+#include "gtest/internal/gtest-internal.h"
+#include "gtest/internal/gtest-port.h"
+
namespace testing {
// Constructs a matcher that matches a const std::string& whose value is
diff --git a/third_party/googletest/src/googletest/src/gtest-port.cc b/third_party/googletest/src/googletest/src/gtest-port.cc
index 53a4d37..d797fe4 100644
--- a/third_party/googletest/src/googletest/src/gtest-port.cc
+++ b/third_party/googletest/src/googletest/src/gtest-port.cc
@@ -27,61 +27,62 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
#include "gtest/internal/gtest-port.h"
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
+
#include <cstdint>
#include <fstream>
#include <memory>
#if GTEST_OS_WINDOWS
-# include <windows.h>
-# include <io.h>
-# include <sys/stat.h>
-# include <map> // Used in ThreadLocal.
-# ifdef _MSC_VER
-# include <crtdbg.h>
-# endif // _MSC_VER
+#include <io.h>
+#include <sys/stat.h>
+#include <windows.h>
+
+#include <map> // Used in ThreadLocal.
+#ifdef _MSC_VER
+#include <crtdbg.h>
+#endif // _MSC_VER
#else
-# include <unistd.h>
+#include <unistd.h>
#endif // GTEST_OS_WINDOWS
#if GTEST_OS_MAC
-# include <mach/mach_init.h>
-# include <mach/task.h>
-# include <mach/vm_map.h>
+#include <mach/mach_init.h>
+#include <mach/task.h>
+#include <mach/vm_map.h>
#endif // GTEST_OS_MAC
#if GTEST_OS_DRAGONFLY || GTEST_OS_FREEBSD || GTEST_OS_GNU_KFREEBSD || \
GTEST_OS_NETBSD || GTEST_OS_OPENBSD
-# include <sys/sysctl.h>
-# if GTEST_OS_DRAGONFLY || GTEST_OS_FREEBSD || GTEST_OS_GNU_KFREEBSD
-# include <sys/user.h>
-# endif
+#include <sys/sysctl.h>
+#if GTEST_OS_DRAGONFLY || GTEST_OS_FREEBSD || GTEST_OS_GNU_KFREEBSD
+#include <sys/user.h>
+#endif
#endif
#if GTEST_OS_QNX
-# include <devctl.h>
-# include <fcntl.h>
-# include <sys/procfs.h>
+#include <devctl.h>
+#include <fcntl.h>
+#include <sys/procfs.h>
#endif // GTEST_OS_QNX
#if GTEST_OS_AIX
-# include <procinfo.h>
-# include <sys/types.h>
+#include <procinfo.h>
+#include <sys/types.h>
#endif // GTEST_OS_AIX
#if GTEST_OS_FUCHSIA
-# include <zircon/process.h>
-# include <zircon/syscalls.h>
+#include <zircon/process.h>
+#include <zircon/syscalls.h>
#endif // GTEST_OS_FUCHSIA
-#include "gtest/gtest-spi.h"
#include "gtest/gtest-message.h"
+#include "gtest/gtest-spi.h"
#include "gtest/internal/gtest-internal.h"
#include "gtest/internal/gtest-string.h"
#include "src/gtest-internal-inl.h"
@@ -89,16 +90,7 @@
namespace testing {
namespace internal {
-#if defined(_MSC_VER) || defined(__BORLANDC__)
-// MSVC and C++Builder do not provide a definition of STDERR_FILENO.
-const int kStdOutFileno = 1;
-const int kStdErrFileno = 2;
-#else
-const int kStdOutFileno = STDOUT_FILENO;
-const int kStdErrFileno = STDERR_FILENO;
-#endif // _MSC_VER
-
-#if GTEST_OS_LINUX
+#if GTEST_OS_LINUX || GTEST_OS_GNU_HURD
namespace {
template <typename T>
@@ -131,8 +123,7 @@
if (status == KERN_SUCCESS) {
// task_threads allocates resources in thread_list and we need to free them
// to avoid leaks.
- vm_deallocate(task,
- reinterpret_cast<vm_address_t>(thread_list),
+ vm_deallocate(task, reinterpret_cast<vm_address_t>(thread_list),
sizeof(thread_t) * thread_count);
return static_cast<size_t>(thread_count);
} else {
@@ -141,7 +132,7 @@
}
#elif GTEST_OS_DRAGONFLY || GTEST_OS_FREEBSD || GTEST_OS_GNU_KFREEBSD || \
- GTEST_OS_NETBSD
+ GTEST_OS_NETBSD
#if GTEST_OS_NETBSD
#undef KERN_PROC
@@ -184,12 +175,12 @@
// we cannot detect it.
size_t GetThreadCount() {
int mib[] = {
- CTL_KERN,
- KERN_PROC,
- KERN_PROC_PID | KERN_PROC_SHOW_THREADS,
- getpid(),
- sizeof(struct kinfo_proc),
- 0,
+ CTL_KERN,
+ KERN_PROC,
+ KERN_PROC_PID | KERN_PROC_SHOW_THREADS,
+ getpid(),
+ sizeof(struct kinfo_proc),
+ 0,
};
u_int miblen = sizeof(mib) / sizeof(mib[0]);
@@ -210,8 +201,7 @@
// exclude empty members
size_t nthreads = 0;
for (size_t i = 0; i < size / static_cast<size_t>(mib[4]); i++) {
- if (info[i].p_tid != -1)
- nthreads++;
+ if (info[i].p_tid != -1) nthreads++;
}
return nthreads;
}
@@ -254,13 +244,9 @@
size_t GetThreadCount() {
int dummy_buffer;
size_t avail;
- zx_status_t status = zx_object_get_info(
- zx_process_self(),
- ZX_INFO_PROCESS_THREADS,
- &dummy_buffer,
- 0,
- nullptr,
- &avail);
+ zx_status_t status =
+ zx_object_get_info(zx_process_self(), ZX_INFO_PROCESS_THREADS,
+ &dummy_buffer, 0, nullptr, &avail);
if (status == ZX_OK) {
return avail;
} else {
@@ -280,27 +266,15 @@
#if GTEST_IS_THREADSAFE && GTEST_OS_WINDOWS
-void SleepMilliseconds(int n) {
- ::Sleep(static_cast<DWORD>(n));
-}
+AutoHandle::AutoHandle() : handle_(INVALID_HANDLE_VALUE) {}
-AutoHandle::AutoHandle()
- : handle_(INVALID_HANDLE_VALUE) {}
+AutoHandle::AutoHandle(Handle handle) : handle_(handle) {}
-AutoHandle::AutoHandle(Handle handle)
- : handle_(handle) {}
+AutoHandle::~AutoHandle() { Reset(); }
-AutoHandle::~AutoHandle() {
- Reset();
-}
+AutoHandle::Handle AutoHandle::Get() const { return handle_; }
-AutoHandle::Handle AutoHandle::Get() const {
- return handle_;
-}
-
-void AutoHandle::Reset() {
- Reset(INVALID_HANDLE_VALUE);
-}
+void AutoHandle::Reset() { Reset(INVALID_HANDLE_VALUE); }
void AutoHandle::Reset(HANDLE handle) {
// Resetting with the same handle we already own is invalid.
@@ -312,7 +286,7 @@
} else {
GTEST_CHECK_(!IsCloseable())
<< "Resetting a valid handle to itself is likely a programmer error "
- "and thus not allowed.";
+ "and thus not allowed.";
}
}
@@ -322,23 +296,6 @@
return handle_ != nullptr && handle_ != INVALID_HANDLE_VALUE;
}
-Notification::Notification()
- : event_(::CreateEvent(nullptr, // Default security attributes.
- TRUE, // Do not reset automatically.
- FALSE, // Initially unset.
- nullptr)) { // Anonymous event.
- GTEST_CHECK_(event_.Get() != nullptr);
-}
-
-void Notification::Notify() {
- GTEST_CHECK_(::SetEvent(event_.Get()) != FALSE);
-}
-
-void Notification::WaitForNotification() {
- GTEST_CHECK_(
- ::WaitForSingleObject(event_.Get(), INFINITE) == WAIT_OBJECT_0);
-}
-
Mutex::Mutex()
: owner_thread_id_(0),
type_(kDynamic),
@@ -391,25 +348,25 @@
// MemoryIsNotDeallocated memory_is_not_deallocated;
// critical_section_ = new CRITICAL_SECTION;
//
-class MemoryIsNotDeallocated
-{
+class MemoryIsNotDeallocated {
public:
MemoryIsNotDeallocated() : old_crtdbg_flag_(0) {
old_crtdbg_flag_ = _CrtSetDbgFlag(_CRTDBG_REPORT_FLAG);
// Set heap allocation block type to _IGNORE_BLOCK so that MS debug CRT
// doesn't report mem leak if there's no matching deallocation.
- _CrtSetDbgFlag(old_crtdbg_flag_ & ~_CRTDBG_ALLOC_MEM_DF);
+ (void)_CrtSetDbgFlag(old_crtdbg_flag_ & ~_CRTDBG_ALLOC_MEM_DF);
}
~MemoryIsNotDeallocated() {
// Restore the original _CRTDBG_ALLOC_MEM_DF flag
- _CrtSetDbgFlag(old_crtdbg_flag_);
+ (void)_CrtSetDbgFlag(old_crtdbg_flag_);
}
private:
int old_crtdbg_flag_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(MemoryIsNotDeallocated);
+ MemoryIsNotDeallocated(const MemoryIsNotDeallocated&) = delete;
+ MemoryIsNotDeallocated& operator=(const MemoryIsNotDeallocated&) = delete;
};
#endif // _MSC_VER
@@ -435,15 +392,13 @@
::InitializeCriticalSection(critical_section_);
// Updates the critical_section_init_phase_ to 2 to signal
// initialization complete.
- GTEST_CHECK_(::InterlockedCompareExchange(
- &critical_section_init_phase_, 2L, 1L) ==
- 1L);
+ GTEST_CHECK_(::InterlockedCompareExchange(&critical_section_init_phase_,
+ 2L, 1L) == 1L);
break;
case 1:
// Somebody else is already initializing the mutex; spin until they
// are done.
- while (::InterlockedCompareExchange(&critical_section_init_phase_,
- 2L,
+ while (::InterlockedCompareExchange(&critical_section_init_phase_, 2L,
2L) != 2L) {
// Possibly yields the rest of the thread's time slice to other
// threads.
@@ -488,9 +443,7 @@
private:
struct ThreadMainParam {
ThreadMainParam(Runnable* runnable, Notification* thread_can_start)
- : runnable_(runnable),
- thread_can_start_(thread_can_start) {
- }
+ : runnable_(runnable), thread_can_start_(thread_can_start) {}
std::unique_ptr<Runnable> runnable_;
// Does not own.
Notification* thread_can_start_;
@@ -508,20 +461,18 @@
// Prohibit instantiation.
ThreadWithParamSupport();
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ThreadWithParamSupport);
+ ThreadWithParamSupport(const ThreadWithParamSupport&) = delete;
+ ThreadWithParamSupport& operator=(const ThreadWithParamSupport&) = delete;
};
} // namespace
-ThreadWithParamBase::ThreadWithParamBase(Runnable *runnable,
+ThreadWithParamBase::ThreadWithParamBase(Runnable* runnable,
Notification* thread_can_start)
- : thread_(ThreadWithParamSupport::CreateThread(runnable,
- thread_can_start)) {
-}
+ : thread_(
+ ThreadWithParamSupport::CreateThread(runnable, thread_can_start)) {}
-ThreadWithParamBase::~ThreadWithParamBase() {
- Join();
-}
+ThreadWithParamBase::~ThreadWithParamBase() { Join(); }
void ThreadWithParamBase::Join() {
GTEST_CHECK_(::WaitForSingleObject(thread_.Get(), INFINITE) == WAIT_OBJECT_0)
@@ -548,8 +499,10 @@
ThreadIdToThreadLocals::iterator thread_local_pos =
thread_to_thread_locals->find(current_thread);
if (thread_local_pos == thread_to_thread_locals->end()) {
- thread_local_pos = thread_to_thread_locals->insert(
- std::make_pair(current_thread, ThreadLocalValues())).first;
+ thread_local_pos =
+ thread_to_thread_locals
+ ->insert(std::make_pair(current_thread, ThreadLocalValues()))
+ .first;
StartWatcherThreadFor(current_thread);
}
ThreadLocalValues& thread_local_values = thread_local_pos->second;
@@ -577,9 +530,8 @@
ThreadIdToThreadLocals* const thread_to_thread_locals =
GetThreadLocalsMapLocked();
for (ThreadIdToThreadLocals::iterator it =
- thread_to_thread_locals->begin();
- it != thread_to_thread_locals->end();
- ++it) {
+ thread_to_thread_locals->begin();
+ it != thread_to_thread_locals->end(); ++it) {
ThreadLocalValues& thread_local_values = it->second;
ThreadLocalValues::iterator value_pos =
thread_local_values.find(thread_local_instance);
@@ -609,9 +561,8 @@
if (thread_local_pos != thread_to_thread_locals->end()) {
ThreadLocalValues& thread_local_values = thread_local_pos->second;
for (ThreadLocalValues::iterator value_pos =
- thread_local_values.begin();
- value_pos != thread_local_values.end();
- ++value_pos) {
+ thread_local_values.begin();
+ value_pos != thread_local_values.end(); ++value_pos) {
value_holders.push_back(value_pos->second);
}
thread_to_thread_locals->erase(thread_local_pos);
@@ -637,9 +588,8 @@
static void StartWatcherThreadFor(DWORD thread_id) {
// The returned handle will be kept in thread_map and closed by
// watcher_thread in WatcherThreadFunc.
- HANDLE thread = ::OpenThread(SYNCHRONIZE | THREAD_QUERY_INFORMATION,
- FALSE,
- thread_id);
+ HANDLE thread =
+ ::OpenThread(SYNCHRONIZE | THREAD_QUERY_INFORMATION, FALSE, thread_id);
GTEST_CHECK_(thread != nullptr);
// We need to pass a valid thread ID pointer into CreateThread for it
// to work correctly under Win98.
@@ -650,7 +600,8 @@
&ThreadLocalRegistryImpl::WatcherThreadFunc,
reinterpret_cast<LPVOID>(new ThreadIdAndHandle(thread_id, thread)),
CREATE_SUSPENDED, &watcher_thread_id);
- GTEST_CHECK_(watcher_thread != nullptr);
+ GTEST_CHECK_(watcher_thread != nullptr)
+ << "CreateThread failed with error " << ::GetLastError() << ".";
// Give the watcher thread the same priority as ours to avoid being
// blocked by it.
::SetThreadPriority(watcher_thread,
@@ -664,8 +615,7 @@
static DWORD WINAPI WatcherThreadFunc(LPVOID param) {
const ThreadIdAndHandle* tah =
reinterpret_cast<const ThreadIdAndHandle*>(param);
- GTEST_CHECK_(
- ::WaitForSingleObject(tah->second, INFINITE) == WAIT_OBJECT_0);
+ GTEST_CHECK_(::WaitForSingleObject(tah->second, INFINITE) == WAIT_OBJECT_0);
OnThreadExit(tah->first);
::CloseHandle(tah->second);
delete tah;
@@ -689,16 +639,17 @@
};
Mutex ThreadLocalRegistryImpl::mutex_(Mutex::kStaticMutex); // NOLINT
-Mutex ThreadLocalRegistryImpl::thread_map_mutex_(Mutex::kStaticMutex); // NOLINT
+Mutex ThreadLocalRegistryImpl::thread_map_mutex_(
+ Mutex::kStaticMutex); // NOLINT
ThreadLocalValueHolderBase* ThreadLocalRegistry::GetValueOnCurrentThread(
- const ThreadLocalBase* thread_local_instance) {
+ const ThreadLocalBase* thread_local_instance) {
return ThreadLocalRegistryImpl::GetValueOnCurrentThread(
thread_local_instance);
}
void ThreadLocalRegistry::OnThreadLocalDestroyed(
- const ThreadLocalBase* thread_local_instance) {
+ const ThreadLocalBase* thread_local_instance) {
ThreadLocalRegistryImpl::OnThreadLocalDestroyed(thread_local_instance);
}
@@ -786,7 +737,7 @@
bool IsAsciiWhiteSpace(char ch) { return IsInSet(ch, " \f\n\r\t\v"); }
bool IsAsciiWordChar(char ch) {
return ('a' <= ch && ch <= 'z') || ('A' <= ch && ch <= 'Z') ||
- ('0' <= ch && ch <= '9') || ch == '_';
+ ('0' <= ch && ch <= '9') || ch == '_';
}
// Returns true if and only if "\\c" is a supported escape sequence.
@@ -799,17 +750,28 @@
bool AtomMatchesChar(bool escaped, char pattern_char, char ch) {
if (escaped) { // "\\p" where p is pattern_char.
switch (pattern_char) {
- case 'd': return IsAsciiDigit(ch);
- case 'D': return !IsAsciiDigit(ch);
- case 'f': return ch == '\f';
- case 'n': return ch == '\n';
- case 'r': return ch == '\r';
- case 's': return IsAsciiWhiteSpace(ch);
- case 'S': return !IsAsciiWhiteSpace(ch);
- case 't': return ch == '\t';
- case 'v': return ch == '\v';
- case 'w': return IsAsciiWordChar(ch);
- case 'W': return !IsAsciiWordChar(ch);
+ case 'd':
+ return IsAsciiDigit(ch);
+ case 'D':
+ return !IsAsciiDigit(ch);
+ case 'f':
+ return ch == '\f';
+ case 'n':
+ return ch == '\n';
+ case 'r':
+ return ch == '\r';
+ case 's':
+ return IsAsciiWhiteSpace(ch);
+ case 'S':
+ return !IsAsciiWhiteSpace(ch);
+ case 't':
+ return ch == '\t';
+ case 'v':
+ return ch == '\v';
+ case 'w':
+ return IsAsciiWordChar(ch);
+ case 'W':
+ return !IsAsciiWordChar(ch);
}
return IsAsciiPunct(pattern_char) && pattern_char == ch;
}
@@ -820,7 +782,8 @@
// Helper function used by ValidateRegex() to format error messages.
static std::string FormatRegexSyntaxError(const char* regex, int index) {
return (Message() << "Syntax error at index " << index
- << " in simple regular expression \"" << regex << "\": ").GetString();
+ << " in simple regular expression \"" << regex << "\": ")
+ .GetString();
}
// Generates non-fatal failures and returns false if regex is invalid;
@@ -862,12 +825,12 @@
<< "'$' can only appear at the end.";
is_valid = false;
} else if (IsInSet(ch, "()[]{}|")) {
- ADD_FAILURE() << FormatRegexSyntaxError(regex, i)
- << "'" << ch << "' is unsupported.";
+ ADD_FAILURE() << FormatRegexSyntaxError(regex, i) << "'" << ch
+ << "' is unsupported.";
is_valid = false;
} else if (IsRepeat(ch) && !prev_repeatable) {
- ADD_FAILURE() << FormatRegexSyntaxError(regex, i)
- << "'" << ch << "' can only follow a repeatable token.";
+ ADD_FAILURE() << FormatRegexSyntaxError(regex, i) << "'" << ch
+ << "' can only follow a repeatable token.";
is_valid = false;
}
@@ -885,12 +848,10 @@
// characters to be indexable by size_t, in which case the test will
// probably time out anyway. We are fine with this limitation as
// std::string has it too.
-bool MatchRepetitionAndRegexAtHead(
- bool escaped, char c, char repeat, const char* regex,
- const char* str) {
+bool MatchRepetitionAndRegexAtHead(bool escaped, char c, char repeat,
+ const char* regex, const char* str) {
const size_t min_count = (repeat == '+') ? 1 : 0;
- const size_t max_count = (repeat == '?') ? 1 :
- static_cast<size_t>(-1) - 1;
+ const size_t max_count = (repeat == '?') ? 1 : static_cast<size_t>(-1) - 1;
// We cannot call numeric_limits::max() as it conflicts with the
// max() macro on Windows.
@@ -903,8 +864,7 @@
// greedy match.
return true;
}
- if (str[i] == '\0' || !AtomMatchesChar(escaped, c, str[i]))
- return false;
+ if (str[i] == '\0' || !AtomMatchesChar(escaped, c, str[i])) return false;
}
return false;
}
@@ -918,25 +878,23 @@
// "$" only matches the end of a string. Note that regex being
// valid guarantees that there's nothing after "$" in it.
- if (*regex == '$')
- return *str == '\0';
+ if (*regex == '$') return *str == '\0';
// Is the first thing in regex an escape sequence?
const bool escaped = *regex == '\\';
- if (escaped)
- ++regex;
+ if (escaped) ++regex;
if (IsRepeat(regex[1])) {
// MatchRepetitionAndRegexAtHead() calls MatchRegexAtHead(), so
// here's an indirect recursion. It terminates as the regex gets
// shorter in each recursion.
- return MatchRepetitionAndRegexAtHead(
- escaped, regex[0], regex[1], regex + 2, str);
+ return MatchRepetitionAndRegexAtHead(escaped, regex[0], regex[1], regex + 2,
+ str);
} else {
// regex isn't empty, isn't "$", and doesn't start with a
// repetition. We match the first atom of regex with the first
// character of str and recurse.
return (*str != '\0') && AtomMatchesChar(escaped, *regex, *str) &&
- MatchRegexAtHead(regex + 1, str + 1);
+ MatchRegexAtHead(regex + 1, str + 1);
}
}
@@ -951,13 +909,11 @@
bool MatchRegexAnywhere(const char* regex, const char* str) {
if (regex == nullptr || str == nullptr) return false;
- if (*regex == '^')
- return MatchRegexAtHead(regex + 1, str);
+ if (*regex == '^') return MatchRegexAtHead(regex + 1, str);
// A successful match can be anywhere in str.
do {
- if (MatchRegexAtHead(regex, str))
- return true;
+ if (MatchRegexAtHead(regex, str)) return true;
} while (*str++ != '\0');
return false;
}
@@ -1038,8 +994,8 @@
// FormatFileLocation in order to contrast the two functions.
// Note that FormatCompilerIndependentFileLocation() does NOT append colon
// to the file location it produces, unlike FormatFileLocation().
-GTEST_API_ ::std::string FormatCompilerIndependentFileLocation(
- const char* file, int line) {
+GTEST_API_ ::std::string FormatCompilerIndependentFileLocation(const char* file,
+ int line) {
const std::string file_name(file == nullptr ? kUnknownFile : file);
if (line < 0)
@@ -1050,12 +1006,13 @@
GTestLog::GTestLog(GTestLogSeverity severity, const char* file, int line)
: severity_(severity) {
- const char* const marker =
- severity == GTEST_INFO ? "[ INFO ]" :
- severity == GTEST_WARNING ? "[WARNING]" :
- severity == GTEST_ERROR ? "[ ERROR ]" : "[ FATAL ]";
- GetStream() << ::std::endl << marker << " "
- << FormatFileLocation(file, line).c_str() << ": ";
+ const char* const marker = severity == GTEST_INFO ? "[ INFO ]"
+ : severity == GTEST_WARNING ? "[WARNING]"
+ : severity == GTEST_ERROR ? "[ ERROR ]"
+ : "[ FATAL ]";
+ GetStream() << ::std::endl
+ << marker << " " << FormatFileLocation(file, line).c_str()
+ << ": ";
}
// Flushes the buffers and, if severity is GTEST_FATAL, aborts the program.
@@ -1078,27 +1035,26 @@
public:
// The ctor redirects the stream to a temporary file.
explicit CapturedStream(int fd) : fd_(fd), uncaptured_fd_(dup(fd)) {
-# if GTEST_OS_WINDOWS
- char temp_dir_path[MAX_PATH + 1] = { '\0' }; // NOLINT
- char temp_file_path[MAX_PATH + 1] = { '\0' }; // NOLINT
+#if GTEST_OS_WINDOWS
+ char temp_dir_path[MAX_PATH + 1] = {'\0'}; // NOLINT
+ char temp_file_path[MAX_PATH + 1] = {'\0'}; // NOLINT
::GetTempPathA(sizeof(temp_dir_path), temp_dir_path);
- const UINT success = ::GetTempFileNameA(temp_dir_path,
- "gtest_redir",
+ const UINT success = ::GetTempFileNameA(temp_dir_path, "gtest_redir",
0, // Generate unique file name.
temp_file_path);
GTEST_CHECK_(success != 0)
<< "Unable to create a temporary file in " << temp_dir_path;
const int captured_fd = creat(temp_file_path, _S_IREAD | _S_IWRITE);
- GTEST_CHECK_(captured_fd != -1) << "Unable to open temporary file "
- << temp_file_path;
+ GTEST_CHECK_(captured_fd != -1)
+ << "Unable to open temporary file " << temp_file_path;
filename_ = temp_file_path;
-# else
+#else
// There's no guarantee that a test has write access to the current
// directory, so we create the temporary file in a temporary directory.
std::string name_template;
-# if GTEST_OS_LINUX_ANDROID
+#if GTEST_OS_LINUX_ANDROID
// Note: Android applications are expected to call the framework's
// Context.getExternalStorageDirectory() method through JNI to get
// the location of the world-writable SD Card directory. However,
@@ -1111,7 +1067,7 @@
// '/sdcard' and other variants cannot be relied on, as they are not
// guaranteed to be mounted, or may have a delay in mounting.
name_template = "/data/local/tmp/";
-# elif GTEST_OS_IOS
+#elif GTEST_OS_IOS
char user_temp_dir[PATH_MAX + 1];
// Documented alternative to NSTemporaryDirectory() (for obtaining creating
@@ -1132,9 +1088,9 @@
name_template = user_temp_dir;
if (name_template.back() != GTEST_PATH_SEP_[0])
name_template.push_back(GTEST_PATH_SEP_[0]);
-# else
+#else
name_template = "/tmp/";
-# endif
+#endif
name_template.append("gtest_captured_stream.XXXXXX");
// mkstemp() modifies the string bytes in place, and does not go beyond the
@@ -1150,15 +1106,13 @@
<< " for test; does the test have access to the /tmp directory?";
}
filename_ = std::move(name_template);
-# endif // GTEST_OS_WINDOWS
+#endif // GTEST_OS_WINDOWS
fflush(nullptr);
dup2(captured_fd, fd_);
close(captured_fd);
}
- ~CapturedStream() {
- remove(filename_.c_str());
- }
+ ~CapturedStream() { remove(filename_.c_str()); }
std::string GetCapturedString() {
if (uncaptured_fd_ != -1) {
@@ -1185,7 +1139,8 @@
// Name of the temporary file holding the stderr output.
::std::string filename_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(CapturedStream);
+ CapturedStream(const CapturedStream&) = delete;
+ CapturedStream& operator=(const CapturedStream&) = delete;
};
GTEST_DISABLE_MSC_DEPRECATED_POP_()
@@ -1213,6 +1168,15 @@
return content;
}
+#if defined(_MSC_VER) || defined(__BORLANDC__)
+// MSVC and C++Builder do not provide a definition of STDERR_FILENO.
+const int kStdOutFileno = 1;
+const int kStdErrFileno = 2;
+#else
+const int kStdOutFileno = STDOUT_FILENO;
+const int kStdErrFileno = STDERR_FILENO;
+#endif // defined(_MSC_VER) || defined(__BORLANDC__)
+
// Starts capturing stdout.
void CaptureStdout() {
CaptureStream(kStdOutFileno, "stdout", &g_captured_stdout);
@@ -1235,10 +1199,6 @@
#endif // GTEST_HAS_STREAM_REDIRECTION
-
-
-
-
size_t GetFileSize(FILE* file) {
fseek(file, 0, SEEK_END);
return static_cast<size_t>(ftell(file));
@@ -1256,7 +1216,8 @@
// Keeps reading the file until we cannot read further or the
// pre-determined file size is reached.
do {
- bytes_last_read = fread(buffer+bytes_read, 1, file_size-bytes_read, file);
+ bytes_last_read =
+ fread(buffer + bytes_read, 1, file_size - bytes_read, file);
bytes_read += bytes_last_read;
} while (bytes_last_read > 0 && bytes_read < file_size);
@@ -1344,7 +1305,7 @@
// LONG_MAX or LONG_MIN when the input overflows.)
result != long_value
// The parsed value overflows as an int32_t.
- ) {
+ ) {
Message msg;
msg << "WARNING: " << src_text
<< " is expected to be a 32-bit integer, but actually"
@@ -1388,8 +1349,8 @@
}
int32_t result = default_value;
- if (!ParseInt32(Message() << "Environment variable " << env_var,
- string_value, &result)) {
+ if (!ParseInt32(Message() << "Environment variable " << env_var, string_value,
+ &result)) {
printf("The default value %s is used.\n",
(Message() << default_value).GetString().c_str());
fflush(stdout);
@@ -1408,7 +1369,7 @@
// not check that the flag is 'output'
// In essence this checks an env variable called XML_OUTPUT_FILE
// and if it is set we prepend "xml:" to its value, if it not set we return ""
-std::string OutputFlagAlsoCheckEnvVar(){
+std::string OutputFlagAlsoCheckEnvVar() {
std::string default_value_for_output_flag = "";
const char* xml_output_file_env = posix::GetEnv("XML_OUTPUT_FILE");
if (nullptr != xml_output_file_env) {
diff --git a/third_party/googletest/src/googletest/src/gtest-printers.cc b/third_party/googletest/src/googletest/src/gtest-printers.cc
index 1b68fcb..f3976d2 100644
--- a/third_party/googletest/src/googletest/src/gtest-printers.cc
+++ b/third_party/googletest/src/googletest/src/gtest-printers.cc
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
// Google Test - The Google C++ Testing and Mocking Framework
//
// This file implements a universal value printer that can print a
@@ -101,7 +100,7 @@
PrintByteSegmentInObjectTo(obj_bytes, 0, kChunkSize, os);
*os << " ... ";
// Rounds up to 2-byte boundary.
- const size_t resume_pos = (count - kChunkSize + 1)/2*2;
+ const size_t resume_pos = (count - kChunkSize + 1) / 2 * 2;
PrintByteSegmentInObjectTo(obj_bytes, resume_pos, count - resume_pos, os);
}
*os << ">";
@@ -136,11 +135,7 @@
// - as is if it's a printable ASCII (e.g. 'a', '2', ' '),
// - as a hexadecimal escape sequence (e.g. '\x7F'), or
// - as a special escape sequence (e.g. '\r', '\n').
-enum CharFormat {
- kAsIs,
- kHexEscape,
- kSpecialEscape
-};
+enum CharFormat { kAsIs, kHexEscape, kSpecialEscape };
// Returns true if c is a printable ASCII character. We test the
// value of c directly instead of calling isprint(), which is buggy on
@@ -213,35 +208,21 @@
}
}
-static const char* GetCharWidthPrefix(char) {
- return "";
-}
+static const char* GetCharWidthPrefix(char) { return ""; }
-static const char* GetCharWidthPrefix(signed char) {
- return "";
-}
+static const char* GetCharWidthPrefix(signed char) { return ""; }
-static const char* GetCharWidthPrefix(unsigned char) {
- return "";
-}
+static const char* GetCharWidthPrefix(unsigned char) { return ""; }
#ifdef __cpp_char8_t
-static const char* GetCharWidthPrefix(char8_t) {
- return "u8";
-}
+static const char* GetCharWidthPrefix(char8_t) { return "u8"; }
#endif
-static const char* GetCharWidthPrefix(char16_t) {
- return "u";
-}
+static const char* GetCharWidthPrefix(char16_t) { return "u"; }
-static const char* GetCharWidthPrefix(char32_t) {
- return "U";
-}
+static const char* GetCharWidthPrefix(char32_t) { return "U"; }
-static const char* GetCharWidthPrefix(wchar_t) {
- return "L";
-}
+static const char* GetCharWidthPrefix(wchar_t) { return "L"; }
// Prints a char c as if it's part of a string literal, escaping it when
// necessary; returns how c was formatted.
@@ -276,8 +257,7 @@
// To aid user debugging, we also print c's code in decimal, unless
// it's 0 (in which case c was printed as '\\0', making the code
// obvious).
- if (c == 0)
- return;
+ if (c == 0) return;
*os << " (" << static_cast<int>(c);
// For more convenience, we print c's code again in hexadecimal,
@@ -304,17 +284,60 @@
<< static_cast<uint32_t>(c);
}
+// gcc/clang __{u,}int128_t
+#if defined(__SIZEOF_INT128__)
+void PrintTo(__uint128_t v, ::std::ostream* os) {
+ if (v == 0) {
+ *os << "0";
+ return;
+ }
+
+ // Buffer large enough for ceil(log10(2^128))==39 and the null terminator
+ char buf[40];
+ char* p = buf + sizeof(buf);
+
+ // Some configurations have a __uint128_t, but no support for built in
+ // division. Do manual long division instead.
+
+ uint64_t high = static_cast<uint64_t>(v >> 64);
+ uint64_t low = static_cast<uint64_t>(v);
+
+ *--p = 0;
+ while (high != 0 || low != 0) {
+ uint64_t high_mod = high % 10;
+ high = high / 10;
+ // This is the long division algorithm specialized for a divisor of 10 and
+ // only two elements.
+ // Notable values:
+ // 2^64 / 10 == 1844674407370955161
+ // 2^64 % 10 == 6
+ const uint64_t carry = 6 * high_mod + low % 10;
+ low = low / 10 + high_mod * 1844674407370955161 + carry / 10;
+
+ char digit = static_cast<char>(carry % 10);
+ *--p = '0' + digit;
+ }
+ *os << p;
+}
+void PrintTo(__int128_t v, ::std::ostream* os) {
+ __uint128_t uv = static_cast<__uint128_t>(v);
+ if (v < 0) {
+ *os << "-";
+ uv = -uv;
+ }
+ PrintTo(uv, os);
+}
+#endif // __SIZEOF_INT128__
+
// Prints the given array of characters to the ostream. CharType must be either
// char, char8_t, char16_t, char32_t, or wchar_t.
// The array starts at begin, the length is len, it may include '\0' characters
// and may not be NUL-terminated.
template <typename CharType>
-GTEST_ATTRIBUTE_NO_SANITIZE_MEMORY_
-GTEST_ATTRIBUTE_NO_SANITIZE_ADDRESS_
-GTEST_ATTRIBUTE_NO_SANITIZE_HWADDRESS_
-GTEST_ATTRIBUTE_NO_SANITIZE_THREAD_
-static CharFormat PrintCharsAsStringTo(
- const CharType* begin, size_t len, ostream* os) {
+GTEST_ATTRIBUTE_NO_SANITIZE_MEMORY_ GTEST_ATTRIBUTE_NO_SANITIZE_ADDRESS_
+ GTEST_ATTRIBUTE_NO_SANITIZE_HWADDRESS_
+ GTEST_ATTRIBUTE_NO_SANITIZE_THREAD_ static CharFormat
+ PrintCharsAsStringTo(const CharType* begin, size_t len, ostream* os) {
const char* const quote_prefix = GetCharWidthPrefix(*begin);
*os << quote_prefix << "\"";
bool is_previous_hex = false;
@@ -340,12 +363,11 @@
// Prints a (const) char/wchar_t array of 'len' elements, starting at address
// 'begin'. CharType must be either char or wchar_t.
template <typename CharType>
-GTEST_ATTRIBUTE_NO_SANITIZE_MEMORY_
-GTEST_ATTRIBUTE_NO_SANITIZE_ADDRESS_
-GTEST_ATTRIBUTE_NO_SANITIZE_HWADDRESS_
-GTEST_ATTRIBUTE_NO_SANITIZE_THREAD_
-static void UniversalPrintCharArray(
- const CharType* begin, size_t len, ostream* os) {
+GTEST_ATTRIBUTE_NO_SANITIZE_MEMORY_ GTEST_ATTRIBUTE_NO_SANITIZE_ADDRESS_
+ GTEST_ATTRIBUTE_NO_SANITIZE_HWADDRESS_
+ GTEST_ATTRIBUTE_NO_SANITIZE_THREAD_ static void
+ UniversalPrintCharArray(const CharType* begin, size_t len,
+ ostream* os) {
// The code
// const char kFoo[] = "foo";
// generates an array of 4, not 3, elements, with the last one being '\0'.
@@ -436,28 +458,28 @@
namespace {
bool ContainsUnprintableControlCodes(const char* str, size_t length) {
- const unsigned char *s = reinterpret_cast<const unsigned char *>(str);
+ const unsigned char* s = reinterpret_cast<const unsigned char*>(str);
for (size_t i = 0; i < length; i++) {
unsigned char ch = *s++;
if (std::iscntrl(ch)) {
- switch (ch) {
+ switch (ch) {
case '\t':
case '\n':
case '\r':
break;
default:
return true;
- }
}
+ }
}
return false;
}
-bool IsUTF8TrailByte(unsigned char t) { return 0x80 <= t && t<= 0xbf; }
+bool IsUTF8TrailByte(unsigned char t) { return 0x80 <= t && t <= 0xbf; }
bool IsValidUTF8(const char* str, size_t length) {
- const unsigned char *s = reinterpret_cast<const unsigned char *>(str);
+ const unsigned char* s = reinterpret_cast<const unsigned char*>(str);
for (size_t i = 0; i < length;) {
unsigned char lead = s[i++];
@@ -470,15 +492,13 @@
} else if (lead <= 0xdf && (i + 1) <= length && IsUTF8TrailByte(s[i])) {
++i; // 2-byte character
} else if (0xe0 <= lead && lead <= 0xef && (i + 2) <= length &&
- IsUTF8TrailByte(s[i]) &&
- IsUTF8TrailByte(s[i + 1]) &&
+ IsUTF8TrailByte(s[i]) && IsUTF8TrailByte(s[i + 1]) &&
// check for non-shortest form and surrogate
(lead != 0xe0 || s[i] >= 0xa0) &&
(lead != 0xed || s[i] < 0xa0)) {
i += 2; // 3-byte character
} else if (0xf0 <= lead && lead <= 0xf4 && (i + 3) <= length &&
- IsUTF8TrailByte(s[i]) &&
- IsUTF8TrailByte(s[i + 1]) &&
+ IsUTF8TrailByte(s[i]) && IsUTF8TrailByte(s[i + 1]) &&
IsUTF8TrailByte(s[i + 2]) &&
// check for non-shortest form
(lead != 0xf0 || s[i] >= 0x90) &&
@@ -502,7 +522,7 @@
void PrintStringTo(const ::std::string& s, ostream* os) {
if (PrintCharsAsStringTo(s.data(), s.size(), os) == kHexEscape) {
- if (GTEST_FLAG(print_utf8)) {
+ if (GTEST_FLAG_GET(print_utf8)) {
ConditionalPrintAsText(s.data(), s.size(), os);
}
}
diff --git a/third_party/googletest/src/googletest/src/gtest-test-part.cc b/third_party/googletest/src/googletest/src/gtest-test-part.cc
index a938683..eb7c8d1 100644
--- a/third_party/googletest/src/googletest/src/gtest-test-part.cc
+++ b/third_party/googletest/src/googletest/src/gtest-test-part.cc
@@ -51,13 +51,11 @@
return os << internal::FormatFileLocation(result.file_name(),
result.line_number())
<< " "
- << (result.type() == TestPartResult::kSuccess
- ? "Success"
- : result.type() == TestPartResult::kSkip
- ? "Skipped"
- : result.type() == TestPartResult::kFatalFailure
- ? "Fatal failure"
- : "Non-fatal failure")
+ << (result.type() == TestPartResult::kSuccess ? "Success"
+ : result.type() == TestPartResult::kSkip ? "Skipped"
+ : result.type() == TestPartResult::kFatalFailure
+ ? "Fatal failure"
+ : "Non-fatal failure")
<< ":\n"
<< result.message() << std::endl;
}
@@ -86,8 +84,8 @@
HasNewFatalFailureHelper::HasNewFatalFailureHelper()
: has_new_fatal_failure_(false),
- original_reporter_(GetUnitTestImpl()->
- GetTestPartResultReporterForCurrentThread()) {
+ original_reporter_(
+ GetUnitTestImpl()->GetTestPartResultReporterForCurrentThread()) {
GetUnitTestImpl()->SetTestPartResultReporterForCurrentThread(this);
}
@@ -98,8 +96,7 @@
void HasNewFatalFailureHelper::ReportTestPartResult(
const TestPartResult& result) {
- if (result.fatally_failed())
- has_new_fatal_failure_ = true;
+ if (result.fatally_failed()) has_new_fatal_failure_ = true;
original_reporter_->ReportTestPartResult(result);
}
diff --git a/third_party/googletest/src/googletest/src/gtest-typed-test.cc b/third_party/googletest/src/googletest/src/gtest-typed-test.cc
index c02c3df..a2828b8 100644
--- a/third_party/googletest/src/googletest/src/gtest-typed-test.cc
+++ b/third_party/googletest/src/googletest/src/gtest-typed-test.cc
@@ -27,7 +27,6 @@
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
#include "gtest/gtest-typed-test.h"
#include "gtest/gtest.h"
@@ -38,8 +37,7 @@
// Skips to the first non-space char in str. Returns an empty string if str
// contains only whitespace characters.
static const char* SkipSpaces(const char* str) {
- while (IsSpace(*str))
- str++;
+ while (IsSpace(*str)) str++;
return str;
}
@@ -85,8 +83,7 @@
}
for (RegisteredTestIter it = registered_tests_.begin();
- it != registered_tests_.end();
- ++it) {
+ it != registered_tests_.end(); ++it) {
if (tests.count(it->first) == 0) {
errors << "You forgot to list test " << it->first << ".\n";
}
diff --git a/third_party/googletest/src/googletest/src/gtest.cc b/third_party/googletest/src/googletest/src/gtest.cc
index 21c611a..6f31dd2 100644
--- a/third_party/googletest/src/googletest/src/gtest.cc
+++ b/third_party/googletest/src/googletest/src/gtest.cc
@@ -31,8 +31,6 @@
// The Google C++ Testing and Mocking Framework (Google Test)
#include "gtest/gtest.h"
-#include "gtest/internal/custom/gtest.h"
-#include "gtest/gtest-spi.h"
#include <ctype.h>
#include <stdarg.h>
@@ -46,79 +44,87 @@
#include <chrono> // NOLINT
#include <cmath>
#include <cstdint>
+#include <initializer_list>
#include <iomanip>
+#include <iterator>
#include <limits>
#include <list>
#include <map>
#include <ostream> // NOLINT
#include <sstream>
+#include <unordered_set>
#include <vector>
+#include "gtest/gtest-assertion-result.h"
+#include "gtest/gtest-spi.h"
+#include "gtest/internal/custom/gtest.h"
+
#if GTEST_OS_LINUX
-# include <fcntl.h> // NOLINT
-# include <limits.h> // NOLINT
-# include <sched.h> // NOLINT
+#include <fcntl.h> // NOLINT
+#include <limits.h> // NOLINT
+#include <sched.h> // NOLINT
// Declares vsnprintf(). This header is not available on Windows.
-# include <strings.h> // NOLINT
-# include <sys/mman.h> // NOLINT
-# include <sys/time.h> // NOLINT
-# include <unistd.h> // NOLINT
-# include <string>
+#include <strings.h> // NOLINT
+#include <sys/mman.h> // NOLINT
+#include <sys/time.h> // NOLINT
+#include <unistd.h> // NOLINT
+
+#include <string>
#elif GTEST_OS_ZOS
-# include <sys/time.h> // NOLINT
+#include <sys/time.h> // NOLINT
// On z/OS we additionally need strings.h for strcasecmp.
-# include <strings.h> // NOLINT
+#include <strings.h> // NOLINT
#elif GTEST_OS_WINDOWS_MOBILE // We are on Windows CE.
-# include <windows.h> // NOLINT
-# undef min
+#include <windows.h> // NOLINT
+#undef min
#elif GTEST_OS_WINDOWS // We are on Windows proper.
-# include <windows.h> // NOLINT
-# undef min
+#include <windows.h> // NOLINT
+#undef min
#ifdef _MSC_VER
-# include <crtdbg.h> // NOLINT
+#include <crtdbg.h> // NOLINT
#endif
-# include <io.h> // NOLINT
-# include <sys/timeb.h> // NOLINT
-# include <sys/types.h> // NOLINT
-# include <sys/stat.h> // NOLINT
+#include <io.h> // NOLINT
+#include <sys/stat.h> // NOLINT
+#include <sys/timeb.h> // NOLINT
+#include <sys/types.h> // NOLINT
-# if GTEST_OS_WINDOWS_MINGW
-# include <sys/time.h> // NOLINT
-# endif // GTEST_OS_WINDOWS_MINGW
+#if GTEST_OS_WINDOWS_MINGW
+#include <sys/time.h> // NOLINT
+#endif // GTEST_OS_WINDOWS_MINGW
#else
// cpplint thinks that the header is already included, so we want to
// silence it.
-# include <sys/time.h> // NOLINT
-# include <unistd.h> // NOLINT
+#include <sys/time.h> // NOLINT
+#include <unistd.h> // NOLINT
#endif // GTEST_OS_LINUX
#if GTEST_HAS_EXCEPTIONS
-# include <stdexcept>
+#include <stdexcept>
#endif
#if GTEST_CAN_STREAM_RESULTS_
-# include <arpa/inet.h> // NOLINT
-# include <netdb.h> // NOLINT
-# include <sys/socket.h> // NOLINT
-# include <sys/types.h> // NOLINT
+#include <arpa/inet.h> // NOLINT
+#include <netdb.h> // NOLINT
+#include <sys/socket.h> // NOLINT
+#include <sys/types.h> // NOLINT
#endif
#include "src/gtest-internal-inl.h"
#if GTEST_OS_WINDOWS
-# define vsnprintf _vsnprintf
+#define vsnprintf _vsnprintf
#endif // GTEST_OS_WINDOWS
#if GTEST_OS_MAC
@@ -131,7 +137,10 @@
#include "absl/debugging/failure_signal_handler.h"
#include "absl/debugging/stacktrace.h"
#include "absl/debugging/symbolize.h"
+#include "absl/flags/parse.h"
+#include "absl/flags/usage.h"
#include "absl/strings/str_cat.h"
+#include "absl/strings/str_replace.h"
#endif // GTEST_HAS_ABSL
namespace testing {
@@ -177,7 +186,7 @@
// is specified on the command line.
bool g_help_flag = false;
-// Utilty function to Open File for Writing
+// Utility function to Open File for Writing
static FILE* OpenFileForWriting(const std::string& output_file) {
FILE* fileout = nullptr;
FilePath output_file_path(output_file);
@@ -216,28 +225,33 @@
return false;
}
+} // namespace testing
+
GTEST_DEFINE_bool_(
- fail_fast, internal::BoolFromGTestEnv("fail_fast", GetDefaultFailFast()),
+ fail_fast,
+ testing::internal::BoolFromGTestEnv("fail_fast",
+ testing::GetDefaultFailFast()),
"True if and only if a test failure should stop further test execution.");
GTEST_DEFINE_bool_(
also_run_disabled_tests,
- internal::BoolFromGTestEnv("also_run_disabled_tests", false),
+ testing::internal::BoolFromGTestEnv("also_run_disabled_tests", false),
"Run disabled tests too, in addition to the tests normally being run.");
GTEST_DEFINE_bool_(
- break_on_failure, internal::BoolFromGTestEnv("break_on_failure", false),
+ break_on_failure,
+ testing::internal::BoolFromGTestEnv("break_on_failure", false),
"True if and only if a failed assertion should be a debugger "
"break-point.");
GTEST_DEFINE_bool_(catch_exceptions,
- internal::BoolFromGTestEnv("catch_exceptions", true),
+ testing::internal::BoolFromGTestEnv("catch_exceptions",
+ true),
"True if and only if " GTEST_NAME_
" should catch exceptions and treat them as test failures.");
GTEST_DEFINE_string_(
- color,
- internal::StringFromGTestEnv("color", "auto"),
+ color, testing::internal::StringFromGTestEnv("color", "auto"),
"Whether to use colors in the output. Valid values: yes, no, "
"and auto. 'auto' means to use colors if the output is "
"being sent to a terminal and the TERM environment variable "
@@ -245,7 +259,8 @@
GTEST_DEFINE_string_(
filter,
- internal::StringFromGTestEnv("filter", GetDefaultFilter()),
+ testing::internal::StringFromGTestEnv("filter",
+ testing::GetDefaultFilter()),
"A colon-separated list of glob (not regex) patterns "
"for filtering the tests to run, optionally followed by a "
"'-' and a : separated list of negative patterns (tests to "
@@ -254,13 +269,14 @@
GTEST_DEFINE_bool_(
install_failure_signal_handler,
- internal::BoolFromGTestEnv("install_failure_signal_handler", false),
- "If true and supported on the current platform, " GTEST_NAME_ " should "
+ testing::internal::BoolFromGTestEnv("install_failure_signal_handler",
+ false),
+ "If true and supported on the current platform, " GTEST_NAME_
+ " should "
"install a signal handler that dumps debugging information when fatal "
"signals are raised.");
-GTEST_DEFINE_bool_(list_tests, false,
- "List all tests without running them.");
+GTEST_DEFINE_bool_(list_tests, false, "List all tests without running them.");
// The net priority order after flag processing is thus:
// --gtest_output command line flag
@@ -269,8 +285,8 @@
// ''
GTEST_DEFINE_string_(
output,
- internal::StringFromGTestEnv("output",
- internal::OutputFlagAlsoCheckEnvVar().c_str()),
+ testing::internal::StringFromGTestEnv(
+ "output", testing::internal::OutputFlagAlsoCheckEnvVar().c_str()),
"A format (defaults to \"xml\" but can be specified to be \"json\"), "
"optionally followed by a colon and an output file name or directory. "
"A directory is indicated by a trailing pathname separator. "
@@ -281,65 +297,79 @@
"digits.");
GTEST_DEFINE_bool_(
- brief, internal::BoolFromGTestEnv("brief", false),
+ brief, testing::internal::BoolFromGTestEnv("brief", false),
"True if only test failures should be displayed in text output.");
-GTEST_DEFINE_bool_(print_time, internal::BoolFromGTestEnv("print_time", true),
+GTEST_DEFINE_bool_(print_time,
+ testing::internal::BoolFromGTestEnv("print_time", true),
"True if and only if " GTEST_NAME_
" should display elapsed time in text output.");
-GTEST_DEFINE_bool_(print_utf8, internal::BoolFromGTestEnv("print_utf8", true),
+GTEST_DEFINE_bool_(print_utf8,
+ testing::internal::BoolFromGTestEnv("print_utf8", true),
"True if and only if " GTEST_NAME_
" prints UTF8 characters as text.");
GTEST_DEFINE_int32_(
- random_seed,
- internal::Int32FromGTestEnv("random_seed", 0),
+ random_seed, testing::internal::Int32FromGTestEnv("random_seed", 0),
"Random number seed to use when shuffling test orders. Must be in range "
"[1, 99999], or 0 to use a seed based on the current time.");
GTEST_DEFINE_int32_(
- repeat,
- internal::Int32FromGTestEnv("repeat", 1),
+ repeat, testing::internal::Int32FromGTestEnv("repeat", 1),
"How many times to repeat each test. Specify a negative number "
"for repeating forever. Useful for shaking out flaky tests.");
+GTEST_DEFINE_bool_(
+ recreate_environments_when_repeating,
+ testing::internal::BoolFromGTestEnv("recreate_environments_when_repeating",
+ false),
+ "Controls whether global test environments are recreated for each repeat "
+ "of the tests. If set to false the global test environments are only set "
+ "up once, for the first iteration, and only torn down once, for the last. "
+ "Useful for shaking out flaky tests with stable, expensive test "
+ "environments. If --gtest_repeat is set to a negative number, meaning "
+ "there is no last run, the environments will always be recreated to avoid "
+ "leaks.");
+
GTEST_DEFINE_bool_(show_internal_stack_frames, false,
"True if and only if " GTEST_NAME_
" should include internal stack frames when "
"printing test failure stack traces.");
-GTEST_DEFINE_bool_(shuffle, internal::BoolFromGTestEnv("shuffle", false),
+GTEST_DEFINE_bool_(shuffle,
+ testing::internal::BoolFromGTestEnv("shuffle", false),
"True if and only if " GTEST_NAME_
" should randomize tests' order on every run.");
GTEST_DEFINE_int32_(
stack_trace_depth,
- internal::Int32FromGTestEnv("stack_trace_depth", kMaxStackTraceDepth),
+ testing::internal::Int32FromGTestEnv("stack_trace_depth",
+ testing::kMaxStackTraceDepth),
"The maximum number of stack frames to print when an "
"assertion fails. The valid range is 0 through 100, inclusive.");
GTEST_DEFINE_string_(
stream_result_to,
- internal::StringFromGTestEnv("stream_result_to", ""),
+ testing::internal::StringFromGTestEnv("stream_result_to", ""),
"This flag specifies the host name and the port number on which to stream "
"test results. Example: \"localhost:555\". The flag is effective only on "
"Linux.");
GTEST_DEFINE_bool_(
throw_on_failure,
- internal::BoolFromGTestEnv("throw_on_failure", false),
+ testing::internal::BoolFromGTestEnv("throw_on_failure", false),
"When this flag is specified, a failed assertion will throw an exception "
"if exceptions are enabled or exit the program with a non-zero code "
"otherwise. For use with an external test framework.");
#if GTEST_USE_OWN_FLAGFILE_FLAG_
GTEST_DEFINE_string_(
- flagfile,
- internal::StringFromGTestEnv("flagfile", ""),
+ flagfile, testing::internal::StringFromGTestEnv("flagfile", ""),
"This flag specifies the flagfile to read command-line flags from.");
#endif // GTEST_USE_OWN_FLAGFILE_FLAG_
+namespace testing {
namespace internal {
// Generates a random number from [0, range), using a Linear
@@ -348,10 +378,9 @@
uint32_t Random::Generate(uint32_t range) {
// These constants are the same as are used in glibc's rand(3).
// Use wider types than necessary to prevent unsigned overflow diagnostics.
- state_ = static_cast<uint32_t>(1103515245ULL*state_ + 12345U) % kMaxRange;
+ state_ = static_cast<uint32_t>(1103515245ULL * state_ + 12345U) % kMaxRange;
- GTEST_CHECK_(range > 0)
- << "Cannot generate a number in the range [0, 0).";
+ GTEST_CHECK_(range > 0) << "Cannot generate a number in the range [0, 0).";
GTEST_CHECK_(range <= kMaxRange)
<< "Generation of a number in [0, " << range << ") was requested, "
<< "but this can only generate numbers in [0, " << kMaxRange << ").";
@@ -396,32 +425,26 @@
}
// AssertHelper constructor.
-AssertHelper::AssertHelper(TestPartResult::Type type,
- const char* file,
- int line,
- const char* message)
- : data_(new AssertHelperData(type, file, line, message)) {
-}
+AssertHelper::AssertHelper(TestPartResult::Type type, const char* file,
+ int line, const char* message)
+ : data_(new AssertHelperData(type, file, line, message)) {}
-AssertHelper::~AssertHelper() {
- delete data_;
-}
+AssertHelper::~AssertHelper() { delete data_; }
// Message assignment, for assertion streaming support.
void AssertHelper::operator=(const Message& message) const {
- UnitTest::GetInstance()->
- AddTestPartResult(data_->type, data_->file, data_->line,
- AppendUserMessage(data_->message, message),
- UnitTest::GetInstance()->impl()
- ->CurrentOsStackTraceExceptTop(1)
- // Skips the stack frame for this function itself.
- ); // NOLINT
+ UnitTest::GetInstance()->AddTestPartResult(
+ data_->type, data_->file, data_->line,
+ AppendUserMessage(data_->message, message),
+ UnitTest::GetInstance()->impl()->CurrentOsStackTraceExceptTop(1)
+ // Skips the stack frame for this function itself.
+ ); // NOLINT
}
namespace {
// When TEST_P is found without a matching INSTANTIATE_TEST_SUITE_P
-// to creates test cases for it, a syntetic test case is
+// to creates test cases for it, a synthetic test case is
// inserted to report ether an error or a log message.
//
// This configuration bit will likely be removed at some point.
@@ -452,7 +475,6 @@
const bool as_error_;
};
-
} // namespace
std::set<std::string>* GetIgnoredParameterizedTestSuites() {
@@ -496,7 +518,8 @@
"To suppress this error for this test suite, insert the following line "
"(in a non-header) in the namespace it is defined in:"
"\n\n"
- "GTEST_ALLOW_UNINSTANTIATED_PARAMETERIZED_TEST(" + name + ");";
+ "GTEST_ALLOW_UNINSTANTIATED_PARAMETERIZED_TEST(" +
+ name + ");";
std::string full_name = "UninstantiatedParameterizedTestSuite<" + name + ">";
RegisterTest( //
@@ -516,19 +539,18 @@
}
void RegisterTypeParameterizedTestSuiteInstantiation(const char* case_name) {
- GetUnitTestImpl()
- ->type_parameterized_test_registry()
- .RegisterInstantiation(case_name);
+ GetUnitTestImpl()->type_parameterized_test_registry().RegisterInstantiation(
+ case_name);
}
void TypeParameterizedTestSuiteRegistry::RegisterTestSuite(
const char* test_suite_name, CodeLocation code_location) {
suites_.emplace(std::string(test_suite_name),
- TypeParameterizedTestSuiteInfo(code_location));
+ TypeParameterizedTestSuiteInfo(code_location));
}
void TypeParameterizedTestSuiteRegistry::RegisterInstantiation(
- const char* test_suite_name) {
+ const char* test_suite_name) {
auto it = suites_.find(std::string(test_suite_name));
if (it != suites_.end()) {
it->second.instantiated = true;
@@ -606,7 +628,8 @@
// Returns the output format, or "" for normal printed output.
std::string UnitTestOptions::GetOutputFormat() {
- const char* const gtest_output_flag = GTEST_FLAG(output).c_str();
+ std::string s = GTEST_FLAG_GET(output);
+ const char* const gtest_output_flag = s.c_str();
const char* const colon = strchr(gtest_output_flag, ':');
return (colon == nullptr)
? std::string(gtest_output_flag)
@@ -617,19 +640,19 @@
// Returns the name of the requested output file, or the default if none
// was explicitly specified.
std::string UnitTestOptions::GetAbsolutePathToOutputFile() {
- const char* const gtest_output_flag = GTEST_FLAG(output).c_str();
+ std::string s = GTEST_FLAG_GET(output);
+ const char* const gtest_output_flag = s.c_str();
std::string format = GetOutputFormat();
- if (format.empty())
- format = std::string(kDefaultOutputFormat);
+ if (format.empty()) format = std::string(kDefaultOutputFormat);
const char* const colon = strchr(gtest_output_flag, ':');
if (colon == nullptr)
return internal::FilePath::MakeFileName(
- internal::FilePath(
- UnitTest::GetInstance()->original_working_dir()),
- internal::FilePath(kDefaultOutputFile), 0,
- format.c_str()).string();
+ internal::FilePath(
+ UnitTest::GetInstance()->original_working_dir()),
+ internal::FilePath(kDefaultOutputFile), 0, format.c_str())
+ .string();
internal::FilePath output_name(colon + 1);
if (!output_name.IsAbsolutePath())
@@ -637,8 +660,7 @@
internal::FilePath(UnitTest::GetInstance()->original_working_dir()),
internal::FilePath(colon + 1));
- if (!output_name.IsDirectory())
- return output_name.string();
+ if (!output_name.IsDirectory()) return output_name.string();
internal::FilePath result(internal::FilePath::GenerateUniqueFileName(
output_name, internal::GetCurrentExecutableName(),
@@ -699,59 +721,119 @@
return true;
}
+namespace {
+
+bool IsGlobPattern(const std::string& pattern) {
+ return std::any_of(pattern.begin(), pattern.end(),
+ [](const char c) { return c == '?' || c == '*'; });
+}
+
+class UnitTestFilter {
+ public:
+ UnitTestFilter() = default;
+
+ // Constructs a filter from a string of patterns separated by `:`.
+ explicit UnitTestFilter(const std::string& filter) {
+ // By design "" filter matches "" string.
+ std::vector<std::string> all_patterns;
+ SplitString(filter, ':', &all_patterns);
+ const auto exact_match_patterns_begin = std::partition(
+ all_patterns.begin(), all_patterns.end(), &IsGlobPattern);
+
+ glob_patterns_.reserve(static_cast<size_t>(
+ std::distance(all_patterns.begin(), exact_match_patterns_begin)));
+ std::move(all_patterns.begin(), exact_match_patterns_begin,
+ std::inserter(glob_patterns_, glob_patterns_.begin()));
+ std::move(
+ exact_match_patterns_begin, all_patterns.end(),
+ std::inserter(exact_match_patterns_, exact_match_patterns_.begin()));
+ }
+
+ // Returns true if and only if name matches at least one of the patterns in
+ // the filter.
+ bool MatchesName(const std::string& name) const {
+ return exact_match_patterns_.count(name) > 0 ||
+ std::any_of(glob_patterns_.begin(), glob_patterns_.end(),
+ [&name](const std::string& pattern) {
+ return PatternMatchesString(
+ name, pattern.c_str(),
+ pattern.c_str() + pattern.size());
+ });
+ }
+
+ private:
+ std::vector<std::string> glob_patterns_;
+ std::unordered_set<std::string> exact_match_patterns_;
+};
+
+class PositiveAndNegativeUnitTestFilter {
+ public:
+ // Constructs a positive and a negative filter from a string. The string
+ // contains a positive filter optionally followed by a '-' character and a
+ // negative filter. In case only a negative filter is provided the positive
+ // filter will be assumed "*".
+ // A filter is a list of patterns separated by ':'.
+ explicit PositiveAndNegativeUnitTestFilter(const std::string& filter) {
+ std::vector<std::string> positive_and_negative_filters;
+
+ // NOTE: `SplitString` always returns a non-empty container.
+ SplitString(filter, '-', &positive_and_negative_filters);
+ const auto& positive_filter = positive_and_negative_filters.front();
+
+ if (positive_and_negative_filters.size() > 1) {
+ positive_filter_ = UnitTestFilter(
+ positive_filter.empty() ? kUniversalFilter : positive_filter);
+
+ // TODO(b/214626361): Fail on multiple '-' characters
+ // For the moment to preserve old behavior we concatenate the rest of the
+ // string parts with `-` as separator to generate the negative filter.
+ auto negative_filter_string = positive_and_negative_filters[1];
+ for (std::size_t i = 2; i < positive_and_negative_filters.size(); i++)
+ negative_filter_string =
+ negative_filter_string + '-' + positive_and_negative_filters[i];
+ negative_filter_ = UnitTestFilter(negative_filter_string);
+ } else {
+ // In case we don't have a negative filter and positive filter is ""
+ // we do not use kUniversalFilter by design as opposed to when we have a
+ // negative filter.
+ positive_filter_ = UnitTestFilter(positive_filter);
+ }
+ }
+
+ // Returns true if and only if test name (this is generated by appending test
+ // suit name and test name via a '.' character) matches the positive filter
+ // and does not match the negative filter.
+ bool MatchesTest(const std::string& test_suite_name,
+ const std::string& test_name) const {
+ return MatchesName(test_suite_name + "." + test_name);
+ }
+
+ // Returns true if and only if name matches the positive filter and does not
+ // match the negative filter.
+ bool MatchesName(const std::string& name) const {
+ return positive_filter_.MatchesName(name) &&
+ !negative_filter_.MatchesName(name);
+ }
+
+ private:
+ UnitTestFilter positive_filter_;
+ UnitTestFilter negative_filter_;
+};
+} // namespace
+
bool UnitTestOptions::MatchesFilter(const std::string& name_str,
const char* filter) {
- // The filter is a list of patterns separated by colons (:).
- const char* pattern = filter;
- while (true) {
- // Find the bounds of this pattern.
- const char* const next_sep = strchr(pattern, ':');
- const char* const pattern_end =
- next_sep != nullptr ? next_sep : pattern + strlen(pattern);
-
- // Check if this pattern matches name_str.
- if (PatternMatchesString(name_str, pattern, pattern_end)) {
- return true;
- }
-
- // Give up on this pattern. However, if we found a pattern separator (:),
- // advance to the next pattern (skipping over the separator) and restart.
- if (next_sep == nullptr) {
- return false;
- }
- pattern = next_sep + 1;
- }
- return true;
+ return UnitTestFilter(filter).MatchesName(name_str);
}
// Returns true if and only if the user-specified filter matches the test
// suite name and the test name.
bool UnitTestOptions::FilterMatchesTest(const std::string& test_suite_name,
const std::string& test_name) {
- const std::string& full_name = test_suite_name + "." + test_name.c_str();
-
// Split --gtest_filter at '-', if there is one, to separate into
// positive filter and negative filter portions
- const char* const p = GTEST_FLAG(filter).c_str();
- const char* const dash = strchr(p, '-');
- std::string positive;
- std::string negative;
- if (dash == nullptr) {
- positive = GTEST_FLAG(filter).c_str(); // Whole string is a positive filter
- negative = "";
- } else {
- positive = std::string(p, dash); // Everything up to the dash
- negative = std::string(dash + 1); // Everything after the dash
- if (positive.empty()) {
- // Treat '-test1' as the same as '*-test1'
- positive = kUniversalFilter;
- }
- }
-
- // A filter is a colon-separated list of patterns. It matches a
- // test if any pattern in it matches the test.
- return (MatchesFilter(full_name, positive.c_str()) &&
- !MatchesFilter(full_name, negative.c_str()));
+ return PositiveAndNegativeUnitTestFilter(GTEST_FLAG_GET(filter))
+ .MatchesTest(test_suite_name, test_name);
}
#if GTEST_HAS_SEH
@@ -771,7 +853,7 @@
bool should_handle = true;
- if (!GTEST_FLAG(catch_exceptions))
+ if (!GTEST_FLAG_GET(catch_exceptions))
should_handle = false;
else if (exception_code == EXCEPTION_BREAKPOINT)
should_handle = false;
@@ -789,8 +871,7 @@
// results. Intercepts only failures from the current thread.
ScopedFakeTestPartResultReporter::ScopedFakeTestPartResultReporter(
TestPartResultArray* result)
- : intercept_mode_(INTERCEPT_ONLY_CURRENT_THREAD),
- result_(result) {
+ : intercept_mode_(INTERCEPT_ONLY_CURRENT_THREAD), result_(result) {
Init();
}
@@ -799,8 +880,7 @@
// results.
ScopedFakeTestPartResultReporter::ScopedFakeTestPartResultReporter(
InterceptMode intercept_mode, TestPartResultArray* result)
- : intercept_mode_(intercept_mode),
- result_(result) {
+ : intercept_mode_(intercept_mode), result_(result) {
Init();
}
@@ -844,9 +924,7 @@
// from user test code. GetTestTypeId() is guaranteed to always
// return the same value, as it always calls GetTypeId<>() from the
// gtest.cc, which is within the Google Test framework.
-TypeId GetTestTypeId() {
- return GetTypeId<Test>();
-}
+TypeId GetTestTypeId() { return GetTypeId<Test>(); }
// The value of GetTestTypeId() as seen from within the Google Test
// library. This is solely for testing GetTestTypeId().
@@ -861,9 +939,9 @@
const TestPartResultArray& results,
TestPartResult::Type type,
const std::string& substr) {
- const std::string expected(type == TestPartResult::kFatalFailure ?
- "1 fatal failure" :
- "1 non-fatal failure");
+ const std::string expected(type == TestPartResult::kFatalFailure
+ ? "1 fatal failure"
+ : "1 non-fatal failure");
Message msg;
if (results.size() != 1) {
msg << "Expected: " << expected << "\n"
@@ -882,10 +960,10 @@
}
if (strstr(r.message(), substr.c_str()) == nullptr) {
- return AssertionFailure() << "Expected: " << expected << " containing \""
- << substr << "\"\n"
- << " Actual:\n"
- << r;
+ return AssertionFailure()
+ << "Expected: " << expected << " containing \"" << substr << "\"\n"
+ << " Actual:\n"
+ << r;
}
return AssertionSuccess();
@@ -908,7 +986,8 @@
}
DefaultGlobalTestPartResultReporter::DefaultGlobalTestPartResultReporter(
- UnitTestImpl* unit_test) : unit_test_(unit_test) {}
+ UnitTestImpl* unit_test)
+ : unit_test_(unit_test) {}
void DefaultGlobalTestPartResultReporter::ReportTestPartResult(
const TestPartResult& result) {
@@ -917,7 +996,8 @@
}
DefaultPerThreadTestPartResultReporter::DefaultPerThreadTestPartResultReporter(
- UnitTestImpl* unit_test) : unit_test_(unit_test) {}
+ UnitTestImpl* unit_test)
+ : unit_test_(unit_test) {}
void DefaultPerThreadTestPartResultReporter::ReportTestPartResult(
const TestPartResult& result) {
@@ -1024,11 +1104,10 @@
// trace but Bar() and CurrentOsStackTraceExceptTop() won't.
std::string UnitTestImpl::CurrentOsStackTraceExceptTop(int skip_count) {
return os_stack_trace_getter()->CurrentStackTrace(
- static_cast<int>(GTEST_FLAG(stack_trace_depth)),
- skip_count + 1
+ static_cast<int>(GTEST_FLAG_GET(stack_trace_depth)), skip_count + 1
// Skips the user-specified number of frames plus this function
// itself.
- ); // NOLINT
+ ); // NOLINT
}
// A helper class for measuring elapsed times.
@@ -1072,8 +1151,7 @@
const int unicode_length =
MultiByteToWideChar(CP_ACP, 0, ansi, length, nullptr, 0);
WCHAR* unicode = new WCHAR[unicode_length + 1];
- MultiByteToWideChar(CP_ACP, 0, ansi, length,
- unicode, unicode_length);
+ MultiByteToWideChar(CP_ACP, 0, ansi, length, unicode, unicode_length);
unicode[unicode_length] = 0;
return unicode;
}
@@ -1082,7 +1160,7 @@
// memory using new. The caller is responsible for deleting the return
// value using delete[]. Returns the ANSI string, or NULL if the
// input is NULL.
-const char* String::Utf16ToAnsi(LPCWSTR utf16_str) {
+const char* String::Utf16ToAnsi(LPCWSTR utf16_str) {
if (!utf16_str) return nullptr;
const int ansi_length = WideCharToMultiByte(CP_ACP, 0, utf16_str, -1, nullptr,
0, nullptr, nullptr);
@@ -1101,7 +1179,7 @@
// Unlike strcmp(), this function can handle NULL argument(s). A NULL
// C string is considered different to any non-NULL C string,
// including the empty string.
-bool String::CStringEquals(const char * lhs, const char * rhs) {
+bool String::CStringEquals(const char* lhs, const char* rhs) {
if (lhs == nullptr) return rhs == nullptr;
if (rhs == nullptr) return false;
@@ -1115,11 +1193,10 @@
// encoding, and streams the result to the given Message object.
static void StreamWideCharsToMessage(const wchar_t* wstr, size_t length,
Message* msg) {
- for (size_t i = 0; i != length; ) { // NOLINT
+ for (size_t i = 0; i != length;) { // NOLINT
if (wstr[i] != L'\0') {
*msg << WideStringToUtf8(wstr + i, static_cast<int>(length - i));
- while (i != length && wstr[i] != L'\0')
- i++;
+ while (i != length && wstr[i] != L'\0') i++;
} else {
*msg << '\0';
i++;
@@ -1161,17 +1238,17 @@
// These two overloads allow streaming a wide C string to a Message
// using the UTF-8 encoding.
-Message& Message::operator <<(const wchar_t* wide_c_str) {
+Message& Message::operator<<(const wchar_t* wide_c_str) {
return *this << internal::String::ShowWideCString(wide_c_str);
}
-Message& Message::operator <<(wchar_t* wide_c_str) {
+Message& Message::operator<<(wchar_t* wide_c_str) {
return *this << internal::String::ShowWideCString(wide_c_str);
}
#if GTEST_HAS_STD_WSTRING
// Converts the given wide string to a narrow string using the UTF-8
// encoding, and streams the result to this Message object.
-Message& Message::operator <<(const ::std::wstring& wstr) {
+Message& Message::operator<<(const ::std::wstring& wstr) {
internal::StreamWideCharsToMessage(wstr.c_str(), wstr.length(), this);
return *this;
}
@@ -1183,44 +1260,6 @@
return internal::StringStreamToString(ss_.get());
}
-// AssertionResult constructors.
-// Used in EXPECT_TRUE/FALSE(assertion_result).
-AssertionResult::AssertionResult(const AssertionResult& other)
- : success_(other.success_),
- message_(other.message_.get() != nullptr
- ? new ::std::string(*other.message_)
- : static_cast< ::std::string*>(nullptr)) {}
-
-// Swaps two AssertionResults.
-void AssertionResult::swap(AssertionResult& other) {
- using std::swap;
- swap(success_, other.success_);
- swap(message_, other.message_);
-}
-
-// Returns the assertion's negation. Used with EXPECT/ASSERT_FALSE.
-AssertionResult AssertionResult::operator!() const {
- AssertionResult negation(!success_);
- if (message_.get() != nullptr) negation << *message_;
- return negation;
-}
-
-// Makes a successful assertion result.
-AssertionResult AssertionSuccess() {
- return AssertionResult(true);
-}
-
-// Makes a failed assertion result.
-AssertionResult AssertionFailure() {
- return AssertionResult(false);
-}
-
-// Makes a failed assertion result with the given failure message.
-// Deprecated; use AssertionFailure() << message.
-AssertionResult AssertionFailure(const Message& message) {
- return AssertionFailure() << message;
-}
-
namespace internal {
namespace edit_distance {
@@ -1512,8 +1551,7 @@
AssertionResult EqFailure(const char* lhs_expression,
const char* rhs_expression,
const std::string& lhs_value,
- const std::string& rhs_value,
- bool ignoring_case) {
+ const std::string& rhs_value, bool ignoring_case) {
Message msg;
msg << "Expected equality of these values:";
msg << "\n " << lhs_expression;
@@ -1530,10 +1568,8 @@
}
if (!lhs_value.empty() && !rhs_value.empty()) {
- const std::vector<std::string> lhs_lines =
- SplitEscapedString(lhs_value);
- const std::vector<std::string> rhs_lines =
- SplitEscapedString(rhs_value);
+ const std::vector<std::string> lhs_lines = SplitEscapedString(lhs_value);
+ const std::vector<std::string> rhs_lines = SplitEscapedString(rhs_value);
if (lhs_lines.size() > 1 || rhs_lines.size() > 1) {
msg << "\nWith diff:\n"
<< edit_distance::CreateUnifiedDiff(lhs_lines, rhs_lines);
@@ -1545,27 +1581,21 @@
// Constructs a failure message for Boolean assertions such as EXPECT_TRUE.
std::string GetBoolAssertionFailureMessage(
- const AssertionResult& assertion_result,
- const char* expression_text,
- const char* actual_predicate_value,
- const char* expected_predicate_value) {
+ const AssertionResult& assertion_result, const char* expression_text,
+ const char* actual_predicate_value, const char* expected_predicate_value) {
const char* actual_message = assertion_result.message();
Message msg;
msg << "Value of: " << expression_text
<< "\n Actual: " << actual_predicate_value;
- if (actual_message[0] != '\0')
- msg << " (" << actual_message << ")";
+ if (actual_message[0] != '\0') msg << " (" << actual_message << ")";
msg << "\nExpected: " << expected_predicate_value;
return msg.GetString();
}
// Helper function for implementing ASSERT_NEAR.
-AssertionResult DoubleNearPredFormat(const char* expr1,
- const char* expr2,
- const char* abs_error_expr,
- double val1,
- double val2,
- double abs_error) {
+AssertionResult DoubleNearPredFormat(const char* expr1, const char* expr2,
+ const char* abs_error_expr, double val1,
+ double val2, double abs_error) {
const double diff = fabs(val1 - val2);
if (diff <= abs_error) return AssertionSuccess();
@@ -1595,20 +1625,17 @@
"EXPECT_EQUAL. Consider using EXPECT_DOUBLE_EQ instead.";
}
return AssertionFailure()
- << "The difference between " << expr1 << " and " << expr2
- << " is " << diff << ", which exceeds " << abs_error_expr << ", where\n"
- << expr1 << " evaluates to " << val1 << ",\n"
- << expr2 << " evaluates to " << val2 << ", and\n"
- << abs_error_expr << " evaluates to " << abs_error << ".";
+ << "The difference between " << expr1 << " and " << expr2 << " is "
+ << diff << ", which exceeds " << abs_error_expr << ", where\n"
+ << expr1 << " evaluates to " << val1 << ",\n"
+ << expr2 << " evaluates to " << val2 << ", and\n"
+ << abs_error_expr << " evaluates to " << abs_error << ".";
}
-
// Helper template for implementing FloatLE() and DoubleLE().
template <typename RawType>
-AssertionResult FloatingPointLE(const char* expr1,
- const char* expr2,
- RawType val1,
- RawType val2) {
+AssertionResult FloatingPointLE(const char* expr1, const char* expr2,
+ RawType val1, RawType val2) {
// Returns success if val1 is less than val2,
if (val1 < val2) {
return AssertionSuccess();
@@ -1633,24 +1660,24 @@
<< val2;
return AssertionFailure()
- << "Expected: (" << expr1 << ") <= (" << expr2 << ")\n"
- << " Actual: " << StringStreamToString(&val1_ss) << " vs "
- << StringStreamToString(&val2_ss);
+ << "Expected: (" << expr1 << ") <= (" << expr2 << ")\n"
+ << " Actual: " << StringStreamToString(&val1_ss) << " vs "
+ << StringStreamToString(&val2_ss);
}
} // namespace internal
// Asserts that val1 is less than, or almost equal to, val2. Fails
// otherwise. In particular, it fails if either val1 or val2 is NaN.
-AssertionResult FloatLE(const char* expr1, const char* expr2,
- float val1, float val2) {
+AssertionResult FloatLE(const char* expr1, const char* expr2, float val1,
+ float val2) {
return internal::FloatingPointLE<float>(expr1, expr2, val1, val2);
}
// Asserts that val1 is less than, or almost equal to, val2. Fails
// otherwise. In particular, it fails if either val1 or val2 is NaN.
-AssertionResult DoubleLE(const char* expr1, const char* expr2,
- double val1, double val2) {
+AssertionResult DoubleLE(const char* expr1, const char* expr2, double val1,
+ double val2) {
return internal::FloatingPointLE<double>(expr1, expr2, val1, val2);
}
@@ -1658,62 +1685,51 @@
// The helper function for {ASSERT|EXPECT}_STREQ.
AssertionResult CmpHelperSTREQ(const char* lhs_expression,
- const char* rhs_expression,
- const char* lhs,
+ const char* rhs_expression, const char* lhs,
const char* rhs) {
if (String::CStringEquals(lhs, rhs)) {
return AssertionSuccess();
}
- return EqFailure(lhs_expression,
- rhs_expression,
- PrintToString(lhs),
- PrintToString(rhs),
- false);
+ return EqFailure(lhs_expression, rhs_expression, PrintToString(lhs),
+ PrintToString(rhs), false);
}
// The helper function for {ASSERT|EXPECT}_STRCASEEQ.
AssertionResult CmpHelperSTRCASEEQ(const char* lhs_expression,
- const char* rhs_expression,
- const char* lhs,
+ const char* rhs_expression, const char* lhs,
const char* rhs) {
if (String::CaseInsensitiveCStringEquals(lhs, rhs)) {
return AssertionSuccess();
}
- return EqFailure(lhs_expression,
- rhs_expression,
- PrintToString(lhs),
- PrintToString(rhs),
- true);
+ return EqFailure(lhs_expression, rhs_expression, PrintToString(lhs),
+ PrintToString(rhs), true);
}
// The helper function for {ASSERT|EXPECT}_STRNE.
AssertionResult CmpHelperSTRNE(const char* s1_expression,
- const char* s2_expression,
- const char* s1,
+ const char* s2_expression, const char* s1,
const char* s2) {
if (!String::CStringEquals(s1, s2)) {
return AssertionSuccess();
} else {
- return AssertionFailure() << "Expected: (" << s1_expression << ") != ("
- << s2_expression << "), actual: \""
- << s1 << "\" vs \"" << s2 << "\"";
+ return AssertionFailure()
+ << "Expected: (" << s1_expression << ") != (" << s2_expression
+ << "), actual: \"" << s1 << "\" vs \"" << s2 << "\"";
}
}
// The helper function for {ASSERT|EXPECT}_STRCASENE.
AssertionResult CmpHelperSTRCASENE(const char* s1_expression,
- const char* s2_expression,
- const char* s1,
+ const char* s2_expression, const char* s1,
const char* s2) {
if (!String::CaseInsensitiveCStringEquals(s1, s2)) {
return AssertionSuccess();
} else {
return AssertionFailure()
- << "Expected: (" << s1_expression << ") != ("
- << s2_expression << ") (ignoring case), actual: \""
- << s1 << "\" vs \"" << s2 << "\"";
+ << "Expected: (" << s1_expression << ") != (" << s2_expression
+ << ") (ignoring case), actual: \"" << s1 << "\" vs \"" << s2 << "\"";
}
}
@@ -1741,8 +1757,7 @@
// StringType here can be either ::std::string or ::std::wstring.
template <typename StringType>
-bool IsSubstringPred(const StringType& needle,
- const StringType& haystack) {
+bool IsSubstringPred(const StringType& needle, const StringType& haystack) {
return haystack.find(needle) != StringType::npos;
}
@@ -1751,21 +1766,22 @@
// StringType here can be const char*, const wchar_t*, ::std::string,
// or ::std::wstring.
template <typename StringType>
-AssertionResult IsSubstringImpl(
- bool expected_to_be_substring,
- const char* needle_expr, const char* haystack_expr,
- const StringType& needle, const StringType& haystack) {
+AssertionResult IsSubstringImpl(bool expected_to_be_substring,
+ const char* needle_expr,
+ const char* haystack_expr,
+ const StringType& needle,
+ const StringType& haystack) {
if (IsSubstringPred(needle, haystack) == expected_to_be_substring)
return AssertionSuccess();
const bool is_wide_string = sizeof(needle[0]) > 1;
const char* const begin_string_quote = is_wide_string ? "L\"" : "\"";
return AssertionFailure()
- << "Value of: " << needle_expr << "\n"
- << " Actual: " << begin_string_quote << needle << "\"\n"
- << "Expected: " << (expected_to_be_substring ? "" : "not ")
- << "a substring of " << haystack_expr << "\n"
- << "Which is: " << begin_string_quote << haystack << "\"";
+ << "Value of: " << needle_expr << "\n"
+ << " Actual: " << begin_string_quote << needle << "\"\n"
+ << "Expected: " << (expected_to_be_substring ? "" : "not ")
+ << "a substring of " << haystack_expr << "\n"
+ << "Which is: " << begin_string_quote << haystack << "\"";
}
} // namespace
@@ -1774,52 +1790,52 @@
// substring of haystack (NULL is considered a substring of itself
// only), and return an appropriate error message when they fail.
-AssertionResult IsSubstring(
- const char* needle_expr, const char* haystack_expr,
- const char* needle, const char* haystack) {
+AssertionResult IsSubstring(const char* needle_expr, const char* haystack_expr,
+ const char* needle, const char* haystack) {
return IsSubstringImpl(true, needle_expr, haystack_expr, needle, haystack);
}
-AssertionResult IsSubstring(
- const char* needle_expr, const char* haystack_expr,
- const wchar_t* needle, const wchar_t* haystack) {
+AssertionResult IsSubstring(const char* needle_expr, const char* haystack_expr,
+ const wchar_t* needle, const wchar_t* haystack) {
return IsSubstringImpl(true, needle_expr, haystack_expr, needle, haystack);
}
-AssertionResult IsNotSubstring(
- const char* needle_expr, const char* haystack_expr,
- const char* needle, const char* haystack) {
+AssertionResult IsNotSubstring(const char* needle_expr,
+ const char* haystack_expr, const char* needle,
+ const char* haystack) {
return IsSubstringImpl(false, needle_expr, haystack_expr, needle, haystack);
}
-AssertionResult IsNotSubstring(
- const char* needle_expr, const char* haystack_expr,
- const wchar_t* needle, const wchar_t* haystack) {
+AssertionResult IsNotSubstring(const char* needle_expr,
+ const char* haystack_expr, const wchar_t* needle,
+ const wchar_t* haystack) {
return IsSubstringImpl(false, needle_expr, haystack_expr, needle, haystack);
}
-AssertionResult IsSubstring(
- const char* needle_expr, const char* haystack_expr,
- const ::std::string& needle, const ::std::string& haystack) {
+AssertionResult IsSubstring(const char* needle_expr, const char* haystack_expr,
+ const ::std::string& needle,
+ const ::std::string& haystack) {
return IsSubstringImpl(true, needle_expr, haystack_expr, needle, haystack);
}
-AssertionResult IsNotSubstring(
- const char* needle_expr, const char* haystack_expr,
- const ::std::string& needle, const ::std::string& haystack) {
+AssertionResult IsNotSubstring(const char* needle_expr,
+ const char* haystack_expr,
+ const ::std::string& needle,
+ const ::std::string& haystack) {
return IsSubstringImpl(false, needle_expr, haystack_expr, needle, haystack);
}
#if GTEST_HAS_STD_WSTRING
-AssertionResult IsSubstring(
- const char* needle_expr, const char* haystack_expr,
- const ::std::wstring& needle, const ::std::wstring& haystack) {
+AssertionResult IsSubstring(const char* needle_expr, const char* haystack_expr,
+ const ::std::wstring& needle,
+ const ::std::wstring& haystack) {
return IsSubstringImpl(true, needle_expr, haystack_expr, needle, haystack);
}
-AssertionResult IsNotSubstring(
- const char* needle_expr, const char* haystack_expr,
- const ::std::wstring& needle, const ::std::wstring& haystack) {
+AssertionResult IsNotSubstring(const char* needle_expr,
+ const char* haystack_expr,
+ const ::std::wstring& needle,
+ const ::std::wstring& haystack) {
return IsSubstringImpl(false, needle_expr, haystack_expr, needle, haystack);
}
#endif // GTEST_HAS_STD_WSTRING
@@ -1831,43 +1847,42 @@
namespace {
// Helper function for IsHRESULT{SuccessFailure} predicates
-AssertionResult HRESULTFailureHelper(const char* expr,
- const char* expected,
+AssertionResult HRESULTFailureHelper(const char* expr, const char* expected,
long hr) { // NOLINT
-# if GTEST_OS_WINDOWS_MOBILE || GTEST_OS_WINDOWS_TV_TITLE
+#if GTEST_OS_WINDOWS_MOBILE || GTEST_OS_WINDOWS_TV_TITLE
// Windows CE doesn't support FormatMessage.
const char error_text[] = "";
-# else
+#else
// Looks up the human-readable system message for the HRESULT code
// and since we're not passing any params to FormatMessage, we don't
// want inserts expanded.
- const DWORD kFlags = FORMAT_MESSAGE_FROM_SYSTEM |
- FORMAT_MESSAGE_IGNORE_INSERTS;
+ const DWORD kFlags =
+ FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS;
const DWORD kBufSize = 4096;
// Gets the system's human readable message string for this HRESULT.
- char error_text[kBufSize] = { '\0' };
+ char error_text[kBufSize] = {'\0'};
DWORD message_length = ::FormatMessageA(kFlags,
- 0, // no source, we're asking system
+ 0, // no source, we're asking system
static_cast<DWORD>(hr), // the error
- 0, // no line width restrictions
+ 0, // no line width restrictions
error_text, // output buffer
kBufSize, // buf size
nullptr); // no arguments for inserts
// Trims tailing white space (FormatMessage leaves a trailing CR-LF)
for (; message_length && IsSpace(error_text[message_length - 1]);
- --message_length) {
+ --message_length) {
error_text[message_length - 1] = '\0';
}
-# endif // GTEST_OS_WINDOWS_MOBILE
+#endif // GTEST_OS_WINDOWS_MOBILE
const std::string error_hex("0x" + String::FormatHexInt(hr));
return ::testing::AssertionFailure()
- << "Expected: " << expr << " " << expected << ".\n"
- << " Actual: " << error_hex << " " << error_text << "\n";
+ << "Expected: " << expr << " " << expected << ".\n"
+ << " Actual: " << error_hex << " " << error_text << "\n";
}
} // namespace
@@ -1901,16 +1916,18 @@
// 17 - 21 bits 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
// The maximum code-point a one-byte UTF-8 sequence can represent.
-constexpr uint32_t kMaxCodePoint1 = (static_cast<uint32_t>(1) << 7) - 1;
+constexpr uint32_t kMaxCodePoint1 = (static_cast<uint32_t>(1) << 7) - 1;
// The maximum code-point a two-byte UTF-8 sequence can represent.
constexpr uint32_t kMaxCodePoint2 = (static_cast<uint32_t>(1) << (5 + 6)) - 1;
// The maximum code-point a three-byte UTF-8 sequence can represent.
-constexpr uint32_t kMaxCodePoint3 = (static_cast<uint32_t>(1) << (4 + 2*6)) - 1;
+constexpr uint32_t kMaxCodePoint3 =
+ (static_cast<uint32_t>(1) << (4 + 2 * 6)) - 1;
// The maximum code-point a four-byte UTF-8 sequence can represent.
-constexpr uint32_t kMaxCodePoint4 = (static_cast<uint32_t>(1) << (3 + 3*6)) - 1;
+constexpr uint32_t kMaxCodePoint4 =
+ (static_cast<uint32_t>(1) << (3 + 3 * 6)) - 1;
// Chops off the n lowest bits from a bit pattern. Returns the n
// lowest bits. As a side effect, the original bit pattern will be
@@ -1935,7 +1952,7 @@
char str[5]; // Big enough for the largest valid code point.
if (code_point <= kMaxCodePoint1) {
str[1] = '\0';
- str[0] = static_cast<char>(code_point); // 0xxxxxxx
+ str[0] = static_cast<char>(code_point); // 0xxxxxxx
} else if (code_point <= kMaxCodePoint2) {
str[2] = '\0';
str[1] = static_cast<char>(0x80 | ChopLowBits(&code_point, 6)); // 10xxxxxx
@@ -1963,8 +1980,8 @@
// and thus should be combined into a single Unicode code point
// using CreateCodePointFromUtf16SurrogatePair.
inline bool IsUtf16SurrogatePair(wchar_t first, wchar_t second) {
- return sizeof(wchar_t) == 2 &&
- (first & 0xFC00) == 0xD800 && (second & 0xFC00) == 0xDC00;
+ return sizeof(wchar_t) == 2 && (first & 0xFC00) == 0xD800 &&
+ (second & 0xFC00) == 0xDC00;
}
// Creates a Unicode code point from UTF16 surrogate pair.
@@ -1995,8 +2012,7 @@
// and contains invalid UTF-16 surrogate pairs, values in those pairs
// will be encoded as individual Unicode characters from Basic Normal Plane.
std::string WideStringToUtf8(const wchar_t* str, int num_chars) {
- if (num_chars == -1)
- num_chars = static_cast<int>(wcslen(str));
+ if (num_chars == -1) num_chars = static_cast<int>(wcslen(str));
::std::stringstream stream;
for (int i = 0; i < num_chars; ++i) {
@@ -2005,8 +2021,8 @@
if (str[i] == L'\0') {
break;
} else if (i + 1 < num_chars && IsUtf16SurrogatePair(str[i], str[i + 1])) {
- unicode_code_point = CreateCodePointFromUtf16SurrogatePair(str[i],
- str[i + 1]);
+ unicode_code_point =
+ CreateCodePointFromUtf16SurrogatePair(str[i], str[i + 1]);
i++;
} else {
unicode_code_point = static_cast<uint32_t>(str[i]);
@@ -2019,7 +2035,7 @@
// Converts a wide C string to an std::string using the UTF-8 encoding.
// NULL will be converted to "(null)".
-std::string String::ShowWideCString(const wchar_t * wide_c_str) {
+std::string String::ShowWideCString(const wchar_t* wide_c_str) {
if (wide_c_str == nullptr) return "(null)";
return internal::WideStringToUtf8(wide_c_str, -1);
@@ -2031,7 +2047,7 @@
// Unlike wcscmp(), this function can handle NULL argument(s). A NULL
// C string is considered different to any non-NULL C string,
// including the empty string.
-bool String::WideCStringEquals(const wchar_t * lhs, const wchar_t * rhs) {
+bool String::WideCStringEquals(const wchar_t* lhs, const wchar_t* rhs) {
if (lhs == nullptr) return rhs == nullptr;
if (rhs == nullptr) return false;
@@ -2041,33 +2057,27 @@
// Helper function for *_STREQ on wide strings.
AssertionResult CmpHelperSTREQ(const char* lhs_expression,
- const char* rhs_expression,
- const wchar_t* lhs,
+ const char* rhs_expression, const wchar_t* lhs,
const wchar_t* rhs) {
if (String::WideCStringEquals(lhs, rhs)) {
return AssertionSuccess();
}
- return EqFailure(lhs_expression,
- rhs_expression,
- PrintToString(lhs),
- PrintToString(rhs),
- false);
+ return EqFailure(lhs_expression, rhs_expression, PrintToString(lhs),
+ PrintToString(rhs), false);
}
// Helper function for *_STRNE on wide strings.
AssertionResult CmpHelperSTRNE(const char* s1_expression,
- const char* s2_expression,
- const wchar_t* s1,
+ const char* s2_expression, const wchar_t* s1,
const wchar_t* s2) {
if (!String::WideCStringEquals(s1, s2)) {
return AssertionSuccess();
}
- return AssertionFailure() << "Expected: (" << s1_expression << ") != ("
- << s2_expression << "), actual: "
- << PrintToString(s1)
- << " vs " << PrintToString(s2);
+ return AssertionFailure()
+ << "Expected: (" << s1_expression << ") != (" << s2_expression
+ << "), actual: " << PrintToString(s1) << " vs " << PrintToString(s2);
}
// Compares two C strings, ignoring case. Returns true if and only if they have
@@ -2076,7 +2086,7 @@
// Unlike strcasecmp(), this function can handle NULL argument(s). A
// NULL C string is considered different to any non-NULL C string,
// including the empty string.
-bool String::CaseInsensitiveCStringEquals(const char * lhs, const char * rhs) {
+bool String::CaseInsensitiveCStringEquals(const char* lhs, const char* rhs) {
if (lhs == nullptr) return rhs == nullptr;
if (rhs == nullptr) return false;
return posix::StrCaseCmp(lhs, rhs) == 0;
@@ -2118,8 +2128,8 @@
// Returns true if and only if str ends with the given suffix, ignoring case.
// Any string is considered to end with an empty suffix.
-bool String::EndsWithCaseInsensitive(
- const std::string& str, const std::string& suffix) {
+bool String::EndsWithCaseInsensitive(const std::string& str,
+ const std::string& suffix) {
const size_t str_len = str.length();
const size_t suffix_len = suffix.length();
return (str_len >= suffix_len) &&
@@ -2202,15 +2212,13 @@
: death_test_count_(0), start_timestamp_(0), elapsed_time_(0) {}
// D'tor.
-TestResult::~TestResult() {
-}
+TestResult::~TestResult() {}
// Returns the i-th test part result among all the results. i can
// range from 0 to total_part_count() - 1. If i is not in that range,
// aborts the program.
const TestPartResult& TestResult::GetTestPartResult(int i) const {
- if (i < 0 || i >= total_part_count())
- internal::posix::Abort();
+ if (i < 0 || i >= total_part_count()) internal::posix::Abort();
return test_part_results_.at(static_cast<size_t>(i));
}
@@ -2218,15 +2226,12 @@
// test_property_count() - 1. If i is not in that range, aborts the
// program.
const TestProperty& TestResult::GetTestProperty(int i) const {
- if (i < 0 || i >= test_property_count())
- internal::posix::Abort();
+ if (i < 0 || i >= test_property_count()) internal::posix::Abort();
return test_properties_.at(static_cast<size_t>(i));
}
// Clears the test part results.
-void TestResult::ClearTestPartResults() {
- test_part_results_.clear();
-}
+void TestResult::ClearTestPartResults() { test_part_results_.clear(); }
// Adds a test part result to the list.
void TestResult::AddTestPartResult(const TestPartResult& test_part_result) {
@@ -2255,15 +2260,8 @@
// The list of reserved attributes used in the <testsuites> element of XML
// output.
static const char* const kReservedTestSuitesAttributes[] = {
- "disabled",
- "errors",
- "failures",
- "name",
- "random_seed",
- "tests",
- "time",
- "timestamp"
-};
+ "disabled", "errors", "failures", "name",
+ "random_seed", "tests", "time", "timestamp"};
// The list of reserved attributes used in the <testsuite> element of XML
// output.
@@ -2273,8 +2271,8 @@
// The list of reserved attributes used in the <testcase> element of XML output.
static const char* const kReservedTestCaseAttributes[] = {
- "classname", "name", "status", "time", "type_param",
- "value_param", "file", "line"};
+ "classname", "name", "status", "time",
+ "type_param", "value_param", "file", "line"};
// Use a slightly different set for allowed output to ensure existing tests can
// still RecordProperty("result") or "RecordProperty(timestamp")
@@ -2336,7 +2334,7 @@
const std::string& property_name,
const std::vector<std::string>& reserved_names) {
if (std::find(reserved_names.begin(), reserved_names.end(), property_name) !=
- reserved_names.end()) {
+ reserved_names.end()) {
ADD_FAILURE() << "Reserved key used in RecordProperty(): " << property_name
<< " (" << FormatWordList(reserved_names)
<< " are reserved by " << GTEST_NAME_ << ")";
@@ -2374,8 +2372,7 @@
// Returns true if and only if the test failed.
bool TestResult::Failed() const {
for (int i = 0; i < total_part_count(); ++i) {
- if (GetTestPartResult(i).failed())
- return true;
+ if (GetTestPartResult(i).failed()) return true;
}
return false;
}
@@ -2416,27 +2413,22 @@
// Creates a Test object.
// The c'tor saves the states of all flags.
-Test::Test()
- : gtest_flag_saver_(new GTEST_FLAG_SAVER_) {
-}
+Test::Test() : gtest_flag_saver_(new GTEST_FLAG_SAVER_) {}
// The d'tor restores the states of all flags. The actual work is
// done by the d'tor of the gtest_flag_saver_ field, and thus not
// visible here.
-Test::~Test() {
-}
+Test::~Test() {}
// Sets up the test fixture.
//
// A sub-class may override this.
-void Test::SetUp() {
-}
+void Test::SetUp() {}
// Tears down the test fixture.
//
// A sub-class may override this.
-void Test::TearDown() {
-}
+void Test::TearDown() {}
// Allows user supplied key value pairs to be recorded for later output.
void Test::RecordProperty(const std::string& key, const std::string& value) {
@@ -2541,8 +2533,8 @@
static std::string* FormatSehExceptionMessage(DWORD exception_code,
const char* location) {
Message message;
- message << "SEH exception with code 0x" << std::setbase(16) <<
- exception_code << std::setbase(10) << " thrown in " << location << ".";
+ message << "SEH exception with code 0x" << std::setbase(16) << exception_code
+ << std::setbase(10) << " thrown in " << location << ".";
return new std::string(message.GetString());
}
@@ -2585,8 +2577,8 @@
// exceptions in the same function. Therefore, we provide a separate
// wrapper function for handling SEH exceptions.)
template <class T, typename Result>
-Result HandleSehExceptionsInMethodIfSupported(
- T* object, Result (T::*method)(), const char* location) {
+Result HandleSehExceptionsInMethodIfSupported(T* object, Result (T::*method)(),
+ const char* location) {
#if GTEST_HAS_SEH
__try {
return (object->*method)();
@@ -2595,8 +2587,8 @@
// We create the exception message on the heap because VC++ prohibits
// creation of objects with destructors on stack in functions using __try
// (see error C2712).
- std::string* exception_message = FormatSehExceptionMessage(
- GetExceptionCode(), location);
+ std::string* exception_message =
+ FormatSehExceptionMessage(GetExceptionCode(), location);
internal::ReportFailureInUnknownLocation(TestPartResult::kFatalFailure,
*exception_message);
delete exception_message;
@@ -2612,8 +2604,8 @@
// exceptions, if they are supported; returns the 0-value for type
// Result in case of an SEH exception.
template <class T, typename Result>
-Result HandleExceptionsInMethodIfSupported(
- T* object, Result (T::*method)(), const char* location) {
+Result HandleExceptionsInMethodIfSupported(T* object, Result (T::*method)(),
+ const char* location) {
// NOTE: The user code can affect the way in which Google Test handles
// exceptions by setting GTEST_FLAG(catch_exceptions), but only before
// RUN_ALL_TESTS() starts. It is technically possible to check the flag
@@ -2623,7 +2615,7 @@
// try {
// // Perform the test method.
// } catch (...) {
- // if (GTEST_FLAG(catch_exceptions))
+ // if (GTEST_FLAG_GET(catch_exceptions))
// // Report the exception as failure.
// else
// throw; // Re-throws the original exception.
@@ -2679,16 +2671,16 @@
// GTEST_SKIP().
if (!HasFatalFailure() && !IsSkipped()) {
impl->os_stack_trace_getter()->UponLeavingGTest();
- internal::HandleExceptionsInMethodIfSupported(
- this, &Test::TestBody, "the test body");
+ internal::HandleExceptionsInMethodIfSupported(this, &Test::TestBody,
+ "the test body");
}
// However, we want to clean up as much as possible. Hence we will
// always call TearDown(), even if SetUp() or the test body has
// failed.
impl->os_stack_trace_getter()->UponLeavingGTest();
- internal::HandleExceptionsInMethodIfSupported(
- this, &Test::TearDown, "TearDown()");
+ internal::HandleExceptionsInMethodIfSupported(this, &Test::TearDown,
+ "TearDown()");
}
// Returns true if and only if the current test has a fatal failure.
@@ -2698,8 +2690,9 @@
// Returns true if and only if the current test has a non-fatal failure.
bool Test::HasNonfatalFailure() {
- return internal::GetUnitTestImpl()->current_test_result()->
- HasNonfatalFailure();
+ return internal::GetUnitTestImpl()
+ ->current_test_result()
+ ->HasNonfatalFailure();
}
// Returns true if and only if the current test was skipped.
@@ -2799,11 +2792,10 @@
// Constructor.
//
// TestNameIs has NO default constructor.
- explicit TestNameIs(const char* name)
- : name_(name) {}
+ explicit TestNameIs(const char* name) : name_(name) {}
// Returns true if and only if the test name of test_info matches name_.
- bool operator()(const TestInfo * test_info) const {
+ bool operator()(const TestInfo* test_info) const {
return test_info && test_info->name() == name_;
}
@@ -2831,20 +2823,20 @@
// Creates the test object, runs it, records its result, and then
// deletes it.
void TestInfo::Run() {
- if (!should_run_) return;
+ TestEventListener* repeater = UnitTest::GetInstance()->listeners().repeater();
+ if (!should_run_) {
+ if (is_disabled_ && matches_filter_) repeater->OnTestDisabled(*this);
+ return;
+ }
// Tells UnitTest where to store test result.
internal::UnitTestImpl* const impl = internal::GetUnitTestImpl();
impl->set_current_test_info(this);
- TestEventListener* repeater = UnitTest::GetInstance()->listeners().repeater();
-
// Notifies the unit test event listeners that a test is about to start.
repeater->OnTestStart(*this);
-
result_.set_start_timestamp(internal::GetTimeInMillis());
internal::Timer timer;
-
impl->os_stack_trace_getter()->UponLeavingGTest();
// Creates the test object.
@@ -3009,11 +3001,18 @@
internal::HandleExceptionsInMethodIfSupported(
this, &TestSuite::RunSetUpTestSuite, "SetUpTestSuite()");
+ const bool skip_all = ad_hoc_test_result().Failed();
+
start_timestamp_ = internal::GetTimeInMillis();
internal::Timer timer;
for (int i = 0; i < total_test_count(); i++) {
- GetMutableTestInfo(i)->Run();
- if (GTEST_FLAG(fail_fast) && GetMutableTestInfo(i)->result()->Failed()) {
+ if (skip_all) {
+ GetMutableTestInfo(i)->Skip();
+ } else {
+ GetMutableTestInfo(i)->Run();
+ }
+ if (GTEST_FLAG_GET(fail_fast) &&
+ GetMutableTestInfo(i)->result()->Failed()) {
for (int j = i + 1; j < total_test_count(); j++) {
GetMutableTestInfo(j)->Skip();
}
@@ -3089,11 +3088,10 @@
//
// FormatCountableNoun(1, "formula", "formuli") returns "1 formula".
// FormatCountableNoun(5, "book", "books") returns "5 books".
-static std::string FormatCountableNoun(int count,
- const char * singular_form,
- const char * plural_form) {
+static std::string FormatCountableNoun(int count, const char* singular_form,
+ const char* plural_form) {
return internal::StreamableToString(count) + " " +
- (count == 1 ? singular_form : plural_form);
+ (count == 1 ? singular_form : plural_form);
}
// Formats the count of tests.
@@ -3110,7 +3108,7 @@
// representation. Both kNonFatalFailure and kFatalFailure are translated
// to "Failure", as the user usually doesn't care about the difference
// between the two when viewing the test result.
-static const char * TestPartResultTypeToString(TestPartResult::Type type) {
+static const char* TestPartResultTypeToString(TestPartResult::Type type) {
switch (type) {
case TestPartResult::kSkip:
return "Skipped\n";
@@ -3137,17 +3135,18 @@
// Prints a TestPartResult to an std::string.
static std::string PrintTestPartResultToString(
const TestPartResult& test_part_result) {
- return (Message()
- << internal::FormatFileLocation(test_part_result.file_name(),
- test_part_result.line_number())
- << " " << TestPartResultTypeToString(test_part_result.type())
- << test_part_result.message()).GetString();
+ return (Message() << internal::FormatFileLocation(
+ test_part_result.file_name(),
+ test_part_result.line_number())
+ << " "
+ << TestPartResultTypeToString(test_part_result.type())
+ << test_part_result.message())
+ .GetString();
}
// Prints a TestPartResult.
static void PrintTestPartResult(const TestPartResult& test_part_result) {
- const std::string& result =
- PrintTestPartResultToString(test_part_result);
+ const std::string& result = PrintTestPartResultToString(test_part_result);
printf("%s\n", result.c_str());
fflush(stdout);
// If the test program runs in Visual Studio or a debugger, the
@@ -3164,8 +3163,8 @@
}
// class PrettyUnitTestResultPrinter
-#if GTEST_OS_WINDOWS && !GTEST_OS_WINDOWS_MOBILE && \
- !GTEST_OS_WINDOWS_PHONE && !GTEST_OS_WINDOWS_RT && !GTEST_OS_WINDOWS_MINGW
+#if GTEST_OS_WINDOWS && !GTEST_OS_WINDOWS_MOBILE && !GTEST_OS_WINDOWS_PHONE && \
+ !GTEST_OS_WINDOWS_RT && !GTEST_OS_WINDOWS_MINGW
// Returns the character attribute for the given color.
static WORD GetColorAttribute(GTestColor color) {
@@ -3176,7 +3175,8 @@
return FOREGROUND_GREEN;
case GTestColor::kYellow:
return FOREGROUND_RED | FOREGROUND_GREEN;
- default: return 0;
+ default:
+ return 0;
}
}
@@ -3232,7 +3232,8 @@
// Returns true if and only if Google Test should use colors in the output.
bool ShouldUseColor(bool stdout_is_tty) {
- const char* const gtest_color = GTEST_FLAG(color).c_str();
+ std::string c = GTEST_FLAG_GET(color);
+ const char* const gtest_color = c.c_str();
if (String::CaseInsensitiveCStringEquals(gtest_color, "auto")) {
#if GTEST_OS_WINDOWS && !GTEST_OS_WINDOWS_MINGW
@@ -3259,9 +3260,9 @@
}
return String::CaseInsensitiveCStringEquals(gtest_color, "yes") ||
- String::CaseInsensitiveCStringEquals(gtest_color, "true") ||
- String::CaseInsensitiveCStringEquals(gtest_color, "t") ||
- String::CStringEquals(gtest_color, "1");
+ String::CaseInsensitiveCStringEquals(gtest_color, "true") ||
+ String::CaseInsensitiveCStringEquals(gtest_color, "t") ||
+ String::CStringEquals(gtest_color, "1");
// We take "yes", "true", "t", and "1" as meaning "yes". If the
// value is neither one of these nor "auto", we treat it as "no" to
// be conservative.
@@ -3273,18 +3274,13 @@
// that would be colored when printed, as can be done on Linux.
GTEST_ATTRIBUTE_PRINTF_(2, 3)
-static void ColoredPrintf(GTestColor color, const char *fmt, ...) {
+static void ColoredPrintf(GTestColor color, const char* fmt, ...) {
va_list args;
va_start(args, fmt);
-#if GTEST_OS_WINDOWS_MOBILE || GTEST_OS_ZOS || GTEST_OS_IOS || \
- GTEST_OS_WINDOWS_PHONE || GTEST_OS_WINDOWS_RT || defined(ESP_PLATFORM)
- const bool use_color = AlwaysFalse();
-#else
static const bool in_color_mode =
ShouldUseColor(posix::IsATTY(posix::FileNo(stdout)) != 0);
const bool use_color = in_color_mode && (color != GTestColor::kDefault);
-#endif // GTEST_OS_WINDOWS_MOBILE || GTEST_OS_ZOS
if (!use_color) {
vprintf(fmt, args);
@@ -3292,8 +3288,8 @@
return;
}
-#if GTEST_OS_WINDOWS && !GTEST_OS_WINDOWS_MOBILE && \
- !GTEST_OS_WINDOWS_PHONE && !GTEST_OS_WINDOWS_RT && !GTEST_OS_WINDOWS_MINGW
+#if GTEST_OS_WINDOWS && !GTEST_OS_WINDOWS_MOBILE && !GTEST_OS_WINDOWS_PHONE && \
+ !GTEST_OS_WINDOWS_RT && !GTEST_OS_WINDOWS_MINGW
const HANDLE stdout_handle = GetStdHandle(STD_OUTPUT_HANDLE);
// Gets the current text color.
@@ -3364,6 +3360,7 @@
#endif // OnTestCaseStart
void OnTestStart(const TestInfo& test_info) override;
+ void OnTestDisabled(const TestInfo& test_info) override;
void OnTestPartResult(const TestPartResult& result) override;
void OnTestEnd(const TestInfo& test_info) override;
@@ -3384,13 +3381,14 @@
static void PrintSkippedTests(const UnitTest& unit_test);
};
- // Fired before each iteration of tests starts.
+// Fired before each iteration of tests starts.
void PrettyUnitTestResultPrinter::OnTestIterationStart(
const UnitTest& unit_test, int iteration) {
- if (GTEST_FLAG(repeat) != 1)
+ if (GTEST_FLAG_GET(repeat) != 1)
printf("\nRepeating all tests (iteration %d) . . .\n\n", iteration + 1);
- const char* const filter = GTEST_FLAG(filter).c_str();
+ std::string f = GTEST_FLAG_GET(filter);
+ const char* const filter = f.c_str();
// Prints the filter if it's not *. This reminds the user that some
// tests may be skipped.
@@ -3406,7 +3404,7 @@
internal::posix::GetEnv(kTestTotalShards));
}
- if (GTEST_FLAG(shuffle)) {
+ if (GTEST_FLAG_GET(shuffle)) {
ColoredPrintf(GTestColor::kYellow,
"Note: Randomizing tests' orders with a seed of %d .\n",
unit_test.random_seed());
@@ -3462,6 +3460,13 @@
fflush(stdout);
}
+void PrettyUnitTestResultPrinter::OnTestDisabled(const TestInfo& test_info) {
+ ColoredPrintf(GTestColor::kYellow, "[ DISABLED ] ");
+ PrintTestName(test_info.test_suite_name(), test_info.name());
+ printf("\n");
+ fflush(stdout);
+}
+
// Called after an assertion failure.
void PrettyUnitTestResultPrinter::OnTestPartResult(
const TestPartResult& result) {
@@ -3486,12 +3491,12 @@
ColoredPrintf(GTestColor::kRed, "[ FAILED ] ");
}
PrintTestName(test_info.test_suite_name(), test_info.name());
- if (test_info.result()->Failed())
- PrintFullTestCommentIfPresent(test_info);
+ if (test_info.result()->Failed()) PrintFullTestCommentIfPresent(test_info);
- if (GTEST_FLAG(print_time)) {
- printf(" (%s ms)\n", internal::StreamableToString(
- test_info.result()->elapsed_time()).c_str());
+ if (GTEST_FLAG_GET(print_time)) {
+ printf(" (%s ms)\n",
+ internal::StreamableToString(test_info.result()->elapsed_time())
+ .c_str());
} else {
printf("\n");
}
@@ -3500,7 +3505,7 @@
#ifndef GTEST_REMOVE_LEGACY_TEST_CASEAPI_
void PrettyUnitTestResultPrinter::OnTestCaseEnd(const TestCase& test_case) {
- if (!GTEST_FLAG(print_time)) return;
+ if (!GTEST_FLAG_GET(print_time)) return;
const std::string counts =
FormatCountableNoun(test_case.test_to_run_count(), "test", "tests");
@@ -3511,7 +3516,7 @@
}
#else
void PrettyUnitTestResultPrinter::OnTestSuiteEnd(const TestSuite& test_suite) {
- if (!GTEST_FLAG(print_time)) return;
+ if (!GTEST_FLAG_GET(print_time)) return;
const std::string counts =
FormatCountableNoun(test_suite.test_to_run_count(), "test", "tests");
@@ -3607,7 +3612,7 @@
printf("%s from %s ran.",
FormatTestCount(unit_test.test_to_run_count()).c_str(),
FormatTestSuiteCount(unit_test.test_suite_to_run_count()).c_str());
- if (GTEST_FLAG(print_time)) {
+ if (GTEST_FLAG_GET(print_time)) {
printf(" (%s ms total)",
internal::StreamableToString(unit_test.elapsed_time()).c_str());
}
@@ -3628,7 +3633,7 @@
}
int num_disabled = unit_test.reportable_disabled_test_count();
- if (num_disabled && !GTEST_FLAG(also_run_disabled_tests)) {
+ if (num_disabled && !GTEST_FLAG_GET(also_run_disabled_tests)) {
if (unit_test.Passed()) {
printf("\n"); // Add a spacer if no FAILURE banner is displayed.
}
@@ -3664,6 +3669,7 @@
#endif // OnTestCaseStart
void OnTestStart(const TestInfo& /*test_info*/) override {}
+ void OnTestDisabled(const TestInfo& /*test_info*/) override {}
void OnTestPartResult(const TestPartResult& result) override;
void OnTestEnd(const TestInfo& test_info) override;
@@ -3700,7 +3706,7 @@
PrintTestName(test_info.test_suite_name(), test_info.name());
PrintFullTestCommentIfPresent(test_info);
- if (GTEST_FLAG(print_time)) {
+ if (GTEST_FLAG_GET(print_time)) {
printf(" (%s ms)\n",
internal::StreamableToString(test_info.result()->elapsed_time())
.c_str());
@@ -3717,7 +3723,7 @@
printf("%s from %s ran.",
FormatTestCount(unit_test.test_to_run_count()).c_str(),
FormatTestSuiteCount(unit_test.test_suite_to_run_count()).c_str());
- if (GTEST_FLAG(print_time)) {
+ if (GTEST_FLAG_GET(print_time)) {
printf(" (%s ms total)",
internal::StreamableToString(unit_test.elapsed_time()).c_str());
}
@@ -3732,7 +3738,7 @@
}
int num_disabled = unit_test.reportable_disabled_test_count();
- if (num_disabled && !GTEST_FLAG(also_run_disabled_tests)) {
+ if (num_disabled && !GTEST_FLAG_GET(also_run_disabled_tests)) {
if (unit_test.Passed()) {
printf("\n"); // Add a spacer if no FAILURE banner is displayed.
}
@@ -3752,7 +3758,7 @@
public:
TestEventRepeater() : forwarding_enabled_(true) {}
~TestEventRepeater() override;
- void Append(TestEventListener *listener);
+ void Append(TestEventListener* listener);
TestEventListener* Release(TestEventListener* listener);
// Controls whether events will be forwarded to listeners_. Set to false
@@ -3770,6 +3776,7 @@
#endif // GTEST_REMOVE_LEGACY_TEST_CASEAPI_
void OnTestSuiteStart(const TestSuite& parameter) override;
void OnTestStart(const TestInfo& test_info) override;
+ void OnTestDisabled(const TestInfo& test_info) override;
void OnTestPartResult(const TestPartResult& result) override;
void OnTestEnd(const TestInfo& test_info) override;
// Legacy API is deprecated but still available
@@ -3789,18 +3796,19 @@
// The list of listeners that receive events.
std::vector<TestEventListener*> listeners_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(TestEventRepeater);
+ TestEventRepeater(const TestEventRepeater&) = delete;
+ TestEventRepeater& operator=(const TestEventRepeater&) = delete;
};
TestEventRepeater::~TestEventRepeater() {
ForEach(listeners_, Delete<TestEventListener>);
}
-void TestEventRepeater::Append(TestEventListener *listener) {
+void TestEventRepeater::Append(TestEventListener* listener) {
listeners_.push_back(listener);
}
-TestEventListener* TestEventRepeater::Release(TestEventListener *listener) {
+TestEventListener* TestEventRepeater::Release(TestEventListener* listener) {
for (size_t i = 0; i < listeners_.size(); ++i) {
if (listeners_[i] == listener) {
listeners_.erase(listeners_.begin() + static_cast<int>(i));
@@ -3813,14 +3821,14 @@
// Since most methods are very similar, use macros to reduce boilerplate.
// This defines a member that forwards the call to all listeners.
-#define GTEST_REPEATER_METHOD_(Name, Type) \
-void TestEventRepeater::Name(const Type& parameter) { \
- if (forwarding_enabled_) { \
- for (size_t i = 0; i < listeners_.size(); i++) { \
- listeners_[i]->Name(parameter); \
- } \
- } \
-}
+#define GTEST_REPEATER_METHOD_(Name, Type) \
+ void TestEventRepeater::Name(const Type& parameter) { \
+ if (forwarding_enabled_) { \
+ for (size_t i = 0; i < listeners_.size(); i++) { \
+ listeners_[i]->Name(parameter); \
+ } \
+ } \
+ }
// This defines a member that forwards the call to all listeners in reverse
// order.
#define GTEST_REVERSE_REPEATER_METHOD_(Name, Type) \
@@ -3840,6 +3848,7 @@
#endif // GTEST_REMOVE_LEGACY_TEST_CASEAPI_
GTEST_REPEATER_METHOD_(OnTestSuiteStart, TestSuite)
GTEST_REPEATER_METHOD_(OnTestStart, TestInfo)
+GTEST_REPEATER_METHOD_(OnTestDisabled, TestInfo)
GTEST_REPEATER_METHOD_(OnTestPartResult, TestPartResult)
GTEST_REPEATER_METHOD_(OnEnvironmentsTearDownStart, UnitTest)
GTEST_REVERSE_REPEATER_METHOD_(OnEnvironmentsSetUpEnd, UnitTest)
@@ -3890,12 +3899,13 @@
private:
// Is c a whitespace character that is normalized to a space character
// when it appears in an XML attribute value?
- static bool IsNormalizableWhitespace(char c) {
- return c == 0x9 || c == 0xA || c == 0xD;
+ static bool IsNormalizableWhitespace(unsigned char c) {
+ return c == '\t' || c == '\n' || c == '\r';
}
// May c appear in a well-formed XML document?
- static bool IsValidXmlCharacter(char c) {
+ // https://www.w3.org/TR/REC-xml/#charsets
+ static bool IsValidXmlCharacter(unsigned char c) {
return IsNormalizableWhitespace(c) || c >= 0x20;
}
@@ -3965,7 +3975,8 @@
// The output file.
const std::string output_file_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(XmlUnitTestResultPrinter);
+ XmlUnitTestResultPrinter(const XmlUnitTestResultPrinter&) = delete;
+ XmlUnitTestResultPrinter& operator=(const XmlUnitTestResultPrinter&) = delete;
};
// Creates a new XmlUnitTestResultPrinter.
@@ -4005,8 +4016,8 @@
// module will consist of ordinary English text.
// If this module is ever modified to produce version 1.1 XML output,
// most invalid characters can be retained using character references.
-std::string XmlUnitTestResultPrinter::EscapeXml(
- const std::string& str, bool is_attribute) {
+std::string XmlUnitTestResultPrinter::EscapeXml(const std::string& str,
+ bool is_attribute) {
Message m;
for (size_t i = 0; i < str.size(); ++i) {
@@ -4034,8 +4045,9 @@
m << '"';
break;
default:
- if (IsValidXmlCharacter(ch)) {
- if (is_attribute && IsNormalizableWhitespace(ch))
+ if (IsValidXmlCharacter(static_cast<unsigned char>(ch))) {
+ if (is_attribute &&
+ IsNormalizableWhitespace(static_cast<unsigned char>(ch)))
m << "&#x" << String::FormatByte(static_cast<unsigned char>(ch))
<< ";";
else
@@ -4056,7 +4068,7 @@
std::string output;
output.reserve(str.size());
for (std::string::const_iterator it = str.begin(); it != str.end(); ++it)
- if (IsValidXmlCharacter(*it))
+ if (IsValidXmlCharacter(static_cast<unsigned char>(*it)))
output.push_back(*it);
return output;
@@ -4064,7 +4076,6 @@
// The following routines generate an XML representation of a UnitTest
// object.
-// GOOGLETEST_CM0009 DO NOT DELETE
//
// This is how Google Test concepts map to the DTD:
//
@@ -4113,12 +4124,12 @@
return "";
// YYYY-MM-DDThh:mm:ss.sss
return StreamableToString(time_struct.tm_year + 1900) + "-" +
- String::FormatIntWidth2(time_struct.tm_mon + 1) + "-" +
- String::FormatIntWidth2(time_struct.tm_mday) + "T" +
- String::FormatIntWidth2(time_struct.tm_hour) + ":" +
- String::FormatIntWidth2(time_struct.tm_min) + ":" +
- String::FormatIntWidth2(time_struct.tm_sec) + "." +
- String::FormatIntWidthN(static_cast<int>(ms % 1000), 3);
+ String::FormatIntWidth2(time_struct.tm_mon + 1) + "-" +
+ String::FormatIntWidth2(time_struct.tm_mday) + "T" +
+ String::FormatIntWidth2(time_struct.tm_hour) + ":" +
+ String::FormatIntWidth2(time_struct.tm_min) + ":" +
+ String::FormatIntWidth2(time_struct.tm_sec) + "." +
+ String::FormatIntWidthN(static_cast<int>(ms % 1000), 3);
}
// Streams an XML CDATA section, escaping invalid CDATA sequences as needed.
@@ -4129,8 +4140,8 @@
for (;;) {
const char* const next_segment = strstr(segment, "]]>");
if (next_segment != nullptr) {
- stream->write(
- segment, static_cast<std::streamsize>(next_segment - segment));
+ stream->write(segment,
+ static_cast<std::streamsize>(next_segment - segment));
*stream << "]]>]]><![CDATA[";
segment = next_segment + strlen("]]>");
} else {
@@ -4142,15 +4153,13 @@
}
void XmlUnitTestResultPrinter::OutputXmlAttribute(
- std::ostream* stream,
- const std::string& element_name,
- const std::string& name,
- const std::string& value) {
+ std::ostream* stream, const std::string& element_name,
+ const std::string& name, const std::string& value) {
const std::vector<std::string>& allowed_names =
GetReservedOutputAttributesForElement(element_name);
GTEST_CHECK_(std::find(allowed_names.begin(), allowed_names.end(), name) !=
- allowed_names.end())
+ allowed_names.end())
<< "Attribute " << name << " is not allowed for element <" << element_name
<< ">.";
@@ -4216,10 +4225,11 @@
OutputXmlAttribute(stream, kTestsuite, "type_param",
test_info.type_param());
}
- if (GTEST_FLAG(list_tests)) {
- OutputXmlAttribute(stream, kTestsuite, "file", test_info.file());
- OutputXmlAttribute(stream, kTestsuite, "line",
- StreamableToString(test_info.line()));
+
+ OutputXmlAttribute(stream, kTestsuite, "file", test_info.file());
+ OutputXmlAttribute(stream, kTestsuite, "line",
+ StreamableToString(test_info.line()));
+ if (GTEST_FLAG_GET(list_tests)) {
*stream << " />\n";
return;
}
@@ -4254,8 +4264,7 @@
internal::FormatCompilerIndependentFileLocation(part.file_name(),
part.line_number());
const std::string summary = location + "\n" + part.summary();
- *stream << " <failure message=\""
- << EscapeXmlAttribute(summary)
+ *stream << " <failure message=\"" << EscapeXmlAttribute(summary)
<< "\" type=\"\">";
const std::string detail = location + "\n" + part.message();
OutputXmlCDataSection(stream, RemoveInvalidXmlCharacters(detail).c_str());
@@ -4295,7 +4304,7 @@
OutputXmlAttribute(stream, kTestsuite, "name", test_suite.name());
OutputXmlAttribute(stream, kTestsuite, "tests",
StreamableToString(test_suite.reportable_test_count()));
- if (!GTEST_FLAG(list_tests)) {
+ if (!GTEST_FLAG_GET(list_tests)) {
OutputXmlAttribute(stream, kTestsuite, "failures",
StreamableToString(test_suite.failed_test_count()));
OutputXmlAttribute(
@@ -4343,7 +4352,7 @@
stream, kTestsuites, "timestamp",
FormatEpochTimeInMillisAsIso8601(unit_test.start_timestamp()));
- if (GTEST_FLAG(shuffle)) {
+ if (GTEST_FLAG_GET(shuffle)) {
OutputXmlAttribute(stream, kTestsuites, "random_seed",
StreamableToString(unit_test.random_seed()));
}
@@ -4396,7 +4405,7 @@
for (int i = 0; i < result.test_property_count(); ++i) {
const TestProperty& property = result.GetTestProperty(i);
attributes << " " << property.key() << "="
- << "\"" << EscapeXmlAttribute(property.value()) << "\"";
+ << "\"" << EscapeXmlAttribute(property.value()) << "\"";
}
return attributes.GetString();
}
@@ -4410,15 +4419,15 @@
return;
}
- *stream << "<" << kProperties << ">\n";
+ *stream << " <" << kProperties << ">\n";
for (int i = 0; i < result.test_property_count(); ++i) {
const TestProperty& property = result.GetTestProperty(i);
- *stream << "<" << kProperty;
+ *stream << " <" << kProperty;
*stream << " name=\"" << EscapeXmlAttribute(property.key()) << "\"";
*stream << " value=\"" << EscapeXmlAttribute(property.value()) << "\"";
*stream << "/>\n";
}
- *stream << "</" << kProperties << ">\n";
+ *stream << " </" << kProperties << ">\n";
}
// End XmlUnitTestResultPrinter
@@ -4442,16 +4451,12 @@
//// streams the attribute as JSON.
static void OutputJsonKey(std::ostream* stream,
const std::string& element_name,
- const std::string& name,
- const std::string& value,
- const std::string& indent,
- bool comma = true);
+ const std::string& name, const std::string& value,
+ const std::string& indent, bool comma = true);
static void OutputJsonKey(std::ostream* stream,
const std::string& element_name,
- const std::string& name,
- int value,
- const std::string& indent,
- bool comma = true);
+ const std::string& name, int value,
+ const std::string& indent, bool comma = true);
// Streams a test suite JSON stanza containing the given test result.
//
@@ -4484,7 +4489,9 @@
// The output file.
const std::string output_file_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(JsonUnitTestResultPrinter);
+ JsonUnitTestResultPrinter(const JsonUnitTestResultPrinter&) = delete;
+ JsonUnitTestResultPrinter& operator=(const JsonUnitTestResultPrinter&) =
+ delete;
};
// Creates a new JsonUnitTestResultPrinter.
@@ -4496,7 +4503,7 @@
}
void JsonUnitTestResultPrinter::OnTestIterationEnd(const UnitTest& unit_test,
- int /*iteration*/) {
+ int /*iteration*/) {
FILE* jsonout = OpenFileForWriting(output_file_);
std::stringstream stream;
PrintJsonUnitTest(&stream, unit_test);
@@ -4562,55 +4569,48 @@
return "";
// YYYY-MM-DDThh:mm:ss
return StreamableToString(time_struct.tm_year + 1900) + "-" +
- String::FormatIntWidth2(time_struct.tm_mon + 1) + "-" +
- String::FormatIntWidth2(time_struct.tm_mday) + "T" +
- String::FormatIntWidth2(time_struct.tm_hour) + ":" +
- String::FormatIntWidth2(time_struct.tm_min) + ":" +
- String::FormatIntWidth2(time_struct.tm_sec) + "Z";
+ String::FormatIntWidth2(time_struct.tm_mon + 1) + "-" +
+ String::FormatIntWidth2(time_struct.tm_mday) + "T" +
+ String::FormatIntWidth2(time_struct.tm_hour) + ":" +
+ String::FormatIntWidth2(time_struct.tm_min) + ":" +
+ String::FormatIntWidth2(time_struct.tm_sec) + "Z";
}
static inline std::string Indent(size_t width) {
return std::string(width, ' ');
}
-void JsonUnitTestResultPrinter::OutputJsonKey(
- std::ostream* stream,
- const std::string& element_name,
- const std::string& name,
- const std::string& value,
- const std::string& indent,
- bool comma) {
+void JsonUnitTestResultPrinter::OutputJsonKey(std::ostream* stream,
+ const std::string& element_name,
+ const std::string& name,
+ const std::string& value,
+ const std::string& indent,
+ bool comma) {
const std::vector<std::string>& allowed_names =
GetReservedOutputAttributesForElement(element_name);
GTEST_CHECK_(std::find(allowed_names.begin(), allowed_names.end(), name) !=
- allowed_names.end())
+ allowed_names.end())
<< "Key \"" << name << "\" is not allowed for value \"" << element_name
<< "\".";
*stream << indent << "\"" << name << "\": \"" << EscapeJson(value) << "\"";
- if (comma)
- *stream << ",\n";
+ if (comma) *stream << ",\n";
}
void JsonUnitTestResultPrinter::OutputJsonKey(
- std::ostream* stream,
- const std::string& element_name,
- const std::string& name,
- int value,
- const std::string& indent,
- bool comma) {
+ std::ostream* stream, const std::string& element_name,
+ const std::string& name, int value, const std::string& indent, bool comma) {
const std::vector<std::string>& allowed_names =
GetReservedOutputAttributesForElement(element_name);
GTEST_CHECK_(std::find(allowed_names.begin(), allowed_names.end(), name) !=
- allowed_names.end())
+ allowed_names.end())
<< "Key \"" << name << "\" is not allowed for value \"" << element_name
<< "\".";
*stream << indent << "\"" << name << "\": " << StreamableToString(value);
- if (comma)
- *stream << ",\n";
+ if (comma) *stream << ",\n";
}
// Streams a test suite JSON stanza containing the given test result.
@@ -4620,7 +4620,7 @@
*stream << Indent(4) << "{\n";
OutputJsonKey(stream, "testsuite", "name", "NonTestSuiteFailure", Indent(6));
OutputJsonKey(stream, "testsuite", "tests", 1, Indent(6));
- if (!GTEST_FLAG(list_tests)) {
+ if (!GTEST_FLAG_GET(list_tests)) {
OutputJsonKey(stream, "testsuite", "failures", 1, Indent(6));
OutputJsonKey(stream, "testsuite", "disabled", 0, Indent(6));
OutputJsonKey(stream, "testsuite", "skipped", 0, Indent(6));
@@ -4674,11 +4674,14 @@
OutputJsonKey(stream, kTestsuite, "type_param", test_info.type_param(),
kIndent);
}
- if (GTEST_FLAG(list_tests)) {
- OutputJsonKey(stream, kTestsuite, "file", test_info.file(), kIndent);
- OutputJsonKey(stream, kTestsuite, "line", test_info.line(), kIndent, false);
+
+ OutputJsonKey(stream, kTestsuite, "file", test_info.file(), kIndent);
+ OutputJsonKey(stream, kTestsuite, "line", test_info.line(), kIndent, false);
+ if (GTEST_FLAG_GET(list_tests)) {
*stream << "\n" << Indent(8) << "}";
return;
+ } else {
+ *stream << ",\n";
}
OutputJsonKey(stream, kTestsuite, "status",
@@ -4710,7 +4713,9 @@
if (part.failed()) {
*stream << ",\n";
if (++failures == 1) {
- *stream << kIndent << "\"" << "failures" << "\": [\n";
+ *stream << kIndent << "\""
+ << "failures"
+ << "\": [\n";
}
const std::string location =
internal::FormatCompilerIndependentFileLocation(part.file_name(),
@@ -4723,8 +4728,7 @@
}
}
- if (failures > 0)
- *stream << "\n" << kIndent << "]";
+ if (failures > 0) *stream << "\n" << kIndent << "]";
*stream << "\n" << Indent(8) << "}";
}
@@ -4738,7 +4742,7 @@
OutputJsonKey(stream, kTestsuite, "name", test_suite.name(), kIndent);
OutputJsonKey(stream, kTestsuite, "tests", test_suite.reportable_test_count(),
kIndent);
- if (!GTEST_FLAG(list_tests)) {
+ if (!GTEST_FLAG_GET(list_tests)) {
OutputJsonKey(stream, kTestsuite, "failures",
test_suite.failed_test_count(), kIndent);
OutputJsonKey(stream, kTestsuite, "disabled",
@@ -4785,7 +4789,7 @@
OutputJsonKey(stream, kTestsuites, "disabled",
unit_test.reportable_disabled_test_count(), kIndent);
OutputJsonKey(stream, kTestsuites, "errors", 0, kIndent);
- if (GTEST_FLAG(shuffle)) {
+ if (GTEST_FLAG_GET(shuffle)) {
OutputJsonKey(stream, kTestsuites, "random_seed", unit_test.random_seed(),
kIndent);
}
@@ -4820,7 +4824,9 @@
OutputJsonTestSuiteForTestResult(stream, unit_test.ad_hoc_test_result());
}
- *stream << "\n" << kIndent << "]\n" << "}\n";
+ *stream << "\n"
+ << kIndent << "]\n"
+ << "}\n";
}
void JsonUnitTestResultPrinter::PrintJsonTestList(
@@ -4855,7 +4861,8 @@
Message attributes;
for (int i = 0; i < result.test_property_count(); ++i) {
const TestProperty& property = result.GetTestProperty(i);
- attributes << ",\n" << indent << "\"" << property.key() << "\": "
+ attributes << ",\n"
+ << indent << "\"" << property.key() << "\": "
<< "\"" << EscapeJson(property.value()) << "\"";
}
return attributes.GetString();
@@ -4895,14 +4902,14 @@
addrinfo hints;
memset(&hints, 0, sizeof(hints));
- hints.ai_family = AF_UNSPEC; // To allow both IPv4 and IPv6 addresses.
+ hints.ai_family = AF_UNSPEC; // To allow both IPv4 and IPv6 addresses.
hints.ai_socktype = SOCK_STREAM;
addrinfo* servinfo = nullptr;
// Use the getaddrinfo() to get a linked list of IP addresses for
// the given host name.
- const int error_num = getaddrinfo(
- host_name_.c_str(), port_num_.c_str(), &hints, &servinfo);
+ const int error_num =
+ getaddrinfo(host_name_.c_str(), port_num_.c_str(), &hints, &servinfo);
if (error_num != 0) {
GTEST_LOG_(WARNING) << "stream_result_to: getaddrinfo() failed: "
<< gai_strerror(error_num);
@@ -4911,8 +4918,8 @@
// Loop through all the results and connect to the first we can.
for (addrinfo* cur_addr = servinfo; sockfd_ == -1 && cur_addr != nullptr;
cur_addr = cur_addr->ai_next) {
- sockfd_ = socket(
- cur_addr->ai_family, cur_addr->ai_socktype, cur_addr->ai_protocol);
+ sockfd_ = socket(cur_addr->ai_family, cur_addr->ai_socktype,
+ cur_addr->ai_protocol);
if (sockfd_ != -1) {
// Connect the client socket to the server socket.
if (connect(sockfd_, cur_addr->ai_addr, cur_addr->ai_addrlen) == -1) {
@@ -4962,7 +4969,7 @@
for (int i = 0; i < raw_stack_size; ++i) {
if (raw_stack[i] == caller_frame &&
- !GTEST_FLAG(show_internal_stack_frames)) {
+ !GTEST_FLAG_GET(show_internal_stack_frames)) {
// Add a marker to the trace and stop adding frames.
absl::StrAppend(&result, kElidedFramesMarker, "\n");
break;
@@ -4981,7 +4988,7 @@
return result;
-#else // !GTEST_HAS_ABSL
+#else // !GTEST_HAS_ABSL
static_cast<void>(max_depth);
static_cast<void>(skip_count);
return "";
@@ -5005,14 +5012,14 @@
class ScopedPrematureExitFile {
public:
explicit ScopedPrematureExitFile(const char* premature_exit_filepath)
- : premature_exit_filepath_(premature_exit_filepath ?
- premature_exit_filepath : "") {
+ : premature_exit_filepath_(
+ premature_exit_filepath ? premature_exit_filepath : "") {
// If a path to the premature-exit file is specified...
if (!premature_exit_filepath_.empty()) {
// create the file with a single "0" character in it. I/O
// errors are ignored as there's nothing better we can do and we
// don't want to fail the test because of this.
- FILE* pfile = posix::FOpen(premature_exit_filepath, "w");
+ FILE* pfile = posix::FOpen(premature_exit_filepath_.c_str(), "w");
fwrite("0", 1, 1, pfile);
fclose(pfile);
}
@@ -5034,7 +5041,8 @@
private:
const std::string premature_exit_filepath_;
- GTEST_DISALLOW_COPY_AND_ASSIGN_(ScopedPrematureExitFile);
+ ScopedPrematureExitFile(const ScopedPrematureExitFile&) = delete;
+ ScopedPrematureExitFile& operator=(const ScopedPrematureExitFile&) = delete;
};
} // namespace internal
@@ -5208,7 +5216,7 @@
// Gets the time of the test program start, in ms from the start of the
// UNIX epoch.
internal::TimeInMillis UnitTest::start_timestamp() const {
- return impl()->start_timestamp();
+ return impl()->start_timestamp();
}
// Gets the elapsed time, in milliseconds.
@@ -5251,9 +5259,7 @@
// Returns the list of event listeners that can be used to track events
// inside Google Test.
-TestEventListeners& UnitTest::listeners() {
- return *impl()->listeners();
-}
+TestEventListeners& UnitTest::listeners() { return *impl()->listeners(); }
// Registers and returns a global test environment. When a test
// program is run, all global test environments will be set-up in the
@@ -5278,12 +5284,11 @@
// assertion macros (e.g. ASSERT_TRUE, EXPECT_EQ, etc) eventually call
// this to report their results. The user code should use the
// assertion macros instead of calling this directly.
-void UnitTest::AddTestPartResult(
- TestPartResult::Type result_type,
- const char* file_name,
- int line_number,
- const std::string& message,
- const std::string& os_stack_trace) GTEST_LOCK_EXCLUDED_(mutex_) {
+void UnitTest::AddTestPartResult(TestPartResult::Type result_type,
+ const char* file_name, int line_number,
+ const std::string& message,
+ const std::string& os_stack_trace)
+ GTEST_LOCK_EXCLUDED_(mutex_) {
Message msg;
msg << message;
@@ -5293,8 +5298,9 @@
for (size_t i = impl_->gtest_trace_stack().size(); i > 0; --i) {
const internal::TraceInfo& trace = impl_->gtest_trace_stack()[i - 1];
- msg << "\n" << internal::FormatFileLocation(trace.file, trace.line)
- << " " << trace.message;
+ msg << "\n"
+ << internal::FormatFileLocation(trace.file, trace.line) << " "
+ << trace.message;
}
}
@@ -5304,8 +5310,8 @@
const TestPartResult result = TestPartResult(
result_type, file_name, line_number, msg.GetString().c_str());
- impl_->GetTestPartResultReporterForCurrentThread()->
- ReportTestPartResult(result);
+ impl_->GetTestPartResultReporterForCurrentThread()->ReportTestPartResult(
+ result);
if (result_type != TestPartResult::kSuccess &&
result_type != TestPartResult::kSkip) {
@@ -5314,7 +5320,7 @@
// in the code (perhaps in order to use Google Test assertions
// with another testing framework) and specify the former on the
// command line for debugging.
- if (GTEST_FLAG(break_on_failure)) {
+ if (GTEST_FLAG_GET(break_on_failure)) {
#if GTEST_OS_WINDOWS && !GTEST_OS_WINDOWS_PHONE && !GTEST_OS_WINDOWS_RT
// Using DebugBreak on Windows allows gtest to still break into a debugger
// when a failure happens and both the --gtest_break_on_failure and
@@ -5331,7 +5337,7 @@
// portability: some debuggers don't correctly trap abort().
*static_cast<volatile int*>(nullptr) = 1;
#endif // GTEST_OS_WINDOWS
- } else if (GTEST_FLAG(throw_on_failure)) {
+ } else if (GTEST_FLAG_GET(throw_on_failure)) {
#if GTEST_HAS_EXCEPTIONS
throw internal::GoogleTestFailureException(result);
#else
@@ -5360,7 +5366,7 @@
// from the main thread.
int UnitTest::Run() {
const bool in_death_test_child_process =
- internal::GTEST_FLAG(internal_run_death_test).length() > 0;
+ GTEST_FLAG_GET(internal_run_death_test).length() > 0;
// Google Test implements this protocol for catching that a test
// program exits before returning control to Google Test:
@@ -5390,7 +5396,7 @@
// Captures the value of GTEST_FLAG(catch_exceptions). This value will be
// used for the duration of the program.
- impl()->set_catch_exceptions(GTEST_FLAG(catch_exceptions));
+ impl()->set_catch_exceptions(GTEST_FLAG_GET(catch_exceptions));
#if GTEST_OS_WINDOWS
// Either the user wants Google Test to catch exceptions thrown by the
@@ -5398,26 +5404,26 @@
// process. In either case the user does not want to see pop-up dialogs
// about crashes - they are expected.
if (impl()->catch_exceptions() || in_death_test_child_process) {
-# if !GTEST_OS_WINDOWS_MOBILE && !GTEST_OS_WINDOWS_PHONE && !GTEST_OS_WINDOWS_RT
+#if !GTEST_OS_WINDOWS_MOBILE && !GTEST_OS_WINDOWS_PHONE && !GTEST_OS_WINDOWS_RT
// SetErrorMode doesn't exist on CE.
SetErrorMode(SEM_FAILCRITICALERRORS | SEM_NOALIGNMENTFAULTEXCEPT |
SEM_NOGPFAULTERRORBOX | SEM_NOOPENFILEERRORBOX);
-# endif // !GTEST_OS_WINDOWS_MOBILE
+#endif // !GTEST_OS_WINDOWS_MOBILE
-# if (defined(_MSC_VER) || GTEST_OS_WINDOWS_MINGW) && !GTEST_OS_WINDOWS_MOBILE
+#if (defined(_MSC_VER) || GTEST_OS_WINDOWS_MINGW) && !GTEST_OS_WINDOWS_MOBILE
// Death test children can be terminated with _abort(). On Windows,
// _abort() can show a dialog with a warning message. This forces the
// abort message to go to stderr instead.
_set_error_mode(_OUT_TO_STDERR);
-# endif
+#endif
-# if defined(_MSC_VER) && !GTEST_OS_WINDOWS_MOBILE
+#if defined(_MSC_VER) && !GTEST_OS_WINDOWS_MOBILE
// In the debug version, Visual Studio pops up a separate dialog
// offering a choice to debug the aborted program. We need to suppress
// this dialog or it will pop up for every EXPECT/ASSERT_DEATH statement
// executed. Google Test will notify the user of any unexpected
// failure via stderr.
- if (!GTEST_FLAG(break_on_failure))
+ if (!GTEST_FLAG_GET(break_on_failure))
_set_abort_behavior(
0x0, // Clear the following flags:
_WRITE_ABORT_MSG | _CALL_REPORTFAULT); // pop-up window, core dump.
@@ -5431,14 +5437,15 @@
_CRTDBG_MODE_FILE | _CRTDBG_MODE_DEBUG);
(void)_CrtSetReportFile(_CRT_ASSERT, _CRTDBG_FILE_STDERR);
}
-# endif
+#endif
}
#endif // GTEST_OS_WINDOWS
return internal::HandleExceptionsInMethodIfSupported(
- impl(),
- &internal::UnitTestImpl::RunAllTests,
- "auxiliary test code (environments or event listeners)") ? 0 : 1;
+ impl(), &internal::UnitTestImpl::RunAllTests,
+ "auxiliary test code (environments or event listeners)")
+ ? 0
+ : 1;
}
// Returns the working directory when the first TEST() or TEST_F() was
@@ -5483,14 +5490,10 @@
}
// Creates an empty UnitTest.
-UnitTest::UnitTest() {
- impl_ = new internal::UnitTestImpl(this);
-}
+UnitTest::UnitTest() { impl_ = new internal::UnitTestImpl(this); }
// Destructor of UnitTest.
-UnitTest::~UnitTest() {
- delete impl_;
-}
+UnitTest::~UnitTest() { delete impl_; }
// Pushes a trace defined by SCOPED_TRACE() on to the per-thread
// Google Test trace stack.
@@ -5501,8 +5504,7 @@
}
// Pops a trace from the per-thread Google Test trace stack.
-void UnitTest::PopGTestTrace()
- GTEST_LOCK_EXCLUDED_(mutex_) {
+void UnitTest::PopGTestTrace() GTEST_LOCK_EXCLUDED_(mutex_) {
internal::MutexLock lock(&mutex_);
impl_->gtest_trace_stack().pop_back();
}
@@ -5599,12 +5601,12 @@
// Initializes event listeners for streaming test results in string form.
// Must not be called before InitGoogleTest.
void UnitTestImpl::ConfigureStreamingOutput() {
- const std::string& target = GTEST_FLAG(stream_result_to);
+ const std::string& target = GTEST_FLAG_GET(stream_result_to);
if (!target.empty()) {
const size_t pos = target.find(':');
if (pos != std::string::npos) {
- listeners()->Append(new StreamingListener(target.substr(0, pos),
- target.substr(pos+1)));
+ listeners()->Append(
+ new StreamingListener(target.substr(0, pos), target.substr(pos + 1)));
} else {
GTEST_LOG_(WARNING) << "unrecognized streaming target \"" << target
<< "\" ignored.";
@@ -5642,7 +5644,7 @@
// to shut down the default XML output before invoking RUN_ALL_TESTS.
ConfigureXmlOutput();
- if (GTEST_FLAG(brief)) {
+ if (GTEST_FLAG_GET(brief)) {
listeners()->SetDefaultResultPrinter(new BriefUnitTestResultPrinter);
}
@@ -5652,7 +5654,7 @@
#endif // GTEST_CAN_STREAM_RESULTS_
#if GTEST_HAS_ABSL
- if (GTEST_FLAG(install_failure_signal_handler)) {
+ if (GTEST_FLAG_GET(install_failure_signal_handler)) {
absl::FailureSignalHandlerOptions options;
absl::InstallFailureSignalHandler(options);
}
@@ -5710,9 +5712,9 @@
auto* const new_test_suite =
new TestSuite(test_suite_name, type_param, set_up_tc, tear_down_tc);
+ const UnitTestFilter death_test_suite_filter(kDeathTestSuiteFilter);
// Is this a death test suite?
- if (internal::UnitTestOptions::MatchesFilter(test_suite_name,
- kDeathTestSuiteFilter)) {
+ if (death_test_suite_filter.MatchesName(test_suite_name)) {
// Yes. Inserts the test suite after the last death test suite
// defined so far. This only works when the test suites haven't
// been shuffled. Otherwise we may end up running a death test
@@ -5749,8 +5751,7 @@
const bool gtest_is_initialized_before_run_all_tests = GTestIsInitialized();
// Do not run any test if the --help flag was specified.
- if (g_help_flag)
- return true;
+ if (g_help_flag) return true;
// Repeats the call to the post-flag parsing initialization in case the
// user didn't call InitGoogleTest.
@@ -5768,11 +5769,11 @@
#if GTEST_HAS_DEATH_TEST
in_subprocess_for_death_test =
(internal_run_death_test_flag_.get() != nullptr);
-# if defined(GTEST_EXTRA_DEATH_TEST_CHILD_SETUP_)
+#if defined(GTEST_EXTRA_DEATH_TEST_CHILD_SETUP_)
if (in_subprocess_for_death_test) {
GTEST_EXTRA_DEATH_TEST_CHILD_SETUP_();
}
-# endif // defined(GTEST_EXTRA_DEATH_TEST_CHILD_SETUP_)
+#endif // defined(GTEST_EXTRA_DEATH_TEST_CHILD_SETUP_)
#endif // GTEST_HAS_DEATH_TEST
const bool should_shard = ShouldShard(kTestTotalShards, kTestShardIndex,
@@ -5780,19 +5781,18 @@
// Compares the full test names with the filter to decide which
// tests to run.
- const bool has_tests_to_run = FilterTests(should_shard
- ? HONOR_SHARDING_PROTOCOL
- : IGNORE_SHARDING_PROTOCOL) > 0;
+ const bool has_tests_to_run =
+ FilterTests(should_shard ? HONOR_SHARDING_PROTOCOL
+ : IGNORE_SHARDING_PROTOCOL) > 0;
// Lists the tests and exits if the --gtest_list_tests flag was specified.
- if (GTEST_FLAG(list_tests)) {
+ if (GTEST_FLAG_GET(list_tests)) {
// This must be called *after* FilterTests() has been called.
ListTestsMatchingFilter();
return true;
}
- random_seed_ = GTEST_FLAG(shuffle) ?
- GetRandomSeedFromFlag(GTEST_FLAG(random_seed)) : 0;
+ random_seed_ = GetRandomSeedFromFlag(GTEST_FLAG_GET(random_seed));
// True if and only if at least one test has failed.
bool failed = false;
@@ -5804,9 +5804,21 @@
// How many times to repeat the tests? We don't want to repeat them
// when we are inside the subprocess of a death test.
- const int repeat = in_subprocess_for_death_test ? 1 : GTEST_FLAG(repeat);
+ const int repeat = in_subprocess_for_death_test ? 1 : GTEST_FLAG_GET(repeat);
+
// Repeats forever if the repeat count is negative.
const bool gtest_repeat_forever = repeat < 0;
+
+ // Should test environments be set up and torn down for each repeat, or only
+ // set up on the first and torn down on the last iteration? If there is no
+ // "last" iteration because the tests will repeat forever, always recreate the
+ // environments to avoid leaks in case one of the environments is using
+ // resources that are external to this process. Without this check there would
+ // be no way to clean up those external resources automatically.
+ const bool recreate_environments_when_repeating =
+ GTEST_FLAG_GET(recreate_environments_when_repeating) ||
+ gtest_repeat_forever;
+
for (int i = 0; gtest_repeat_forever || i != repeat; i++) {
// We want to preserve failures generated by ad-hoc test
// assertions executed before RUN_ALL_TESTS().
@@ -5815,7 +5827,7 @@
Timer timer;
// Shuffles test suites and tests if requested.
- if (has_tests_to_run && GTEST_FLAG(shuffle)) {
+ if (has_tests_to_run && GTEST_FLAG_GET(shuffle)) {
random()->Reseed(static_cast<uint32_t>(random_seed_));
// This should be done before calling OnTestIterationStart(),
// such that a test event listener can see the actual test order
@@ -5828,10 +5840,13 @@
// Runs each test suite if there is at least one test to run.
if (has_tests_to_run) {
- // Sets up all environments beforehand.
- repeater->OnEnvironmentsSetUpStart(*parent_);
- ForEach(environments_, SetUpEnvironment);
- repeater->OnEnvironmentsSetUpEnd(*parent_);
+ // Sets up all environments beforehand. If test environments aren't
+ // recreated for each iteration, only do so on the first iteration.
+ if (i == 0 || recreate_environments_when_repeating) {
+ repeater->OnEnvironmentsSetUpStart(*parent_);
+ ForEach(environments_, SetUpEnvironment);
+ repeater->OnEnvironmentsSetUpEnd(*parent_);
+ }
// Runs the tests only if there was no fatal failure or skip triggered
// during global set-up.
@@ -5853,7 +5868,7 @@
for (int test_index = 0; test_index < total_test_suite_count();
test_index++) {
GetMutableSuiteCase(test_index)->Run();
- if (GTEST_FLAG(fail_fast) &&
+ if (GTEST_FLAG_GET(fail_fast) &&
GetMutableSuiteCase(test_index)->Failed()) {
for (int j = test_index + 1; j < total_test_suite_count(); j++) {
GetMutableSuiteCase(j)->Skip();
@@ -5871,11 +5886,15 @@
}
}
- // Tears down all environments in reverse order afterwards.
- repeater->OnEnvironmentsTearDownStart(*parent_);
- std::for_each(environments_.rbegin(), environments_.rend(),
- TearDownEnvironment);
- repeater->OnEnvironmentsTearDownEnd(*parent_);
+ // Tears down all environments in reverse order afterwards. If test
+ // environments aren't recreated for each iteration, only do so on the
+ // last iteration.
+ if (i == repeat - 1 || recreate_environments_when_repeating) {
+ repeater->OnEnvironmentsTearDownStart(*parent_);
+ std::for_each(environments_.rbegin(), environments_.rend(),
+ TearDownEnvironment);
+ repeater->OnEnvironmentsTearDownEnd(*parent_);
+ }
}
elapsed_time_ = timer.Elapsed();
@@ -5896,7 +5915,7 @@
// (it's always safe to unshuffle the tests).
UnshuffleTests();
- if (GTEST_FLAG(shuffle)) {
+ if (GTEST_FLAG_GET(shuffle)) {
// Picks a new random seed for each iteration.
random_seed_ = GetNextRandomSeed(random_seed_);
}
@@ -5947,8 +5966,7 @@
// an error and exits. If in_subprocess_for_death_test, sharding is
// disabled because it must only be applied to the original test
// process. Otherwise, we could filter out death tests we intended to execute.
-bool ShouldShard(const char* total_shards_env,
- const char* shard_index_env,
+bool ShouldShard(const char* total_shards_env, const char* shard_index_env,
bool in_subprocess_for_death_test) {
if (in_subprocess_for_death_test) {
return false;
@@ -5960,27 +5978,27 @@
if (total_shards == -1 && shard_index == -1) {
return false;
} else if (total_shards == -1 && shard_index != -1) {
- const Message msg = Message()
- << "Invalid environment variables: you have "
- << kTestShardIndex << " = " << shard_index
- << ", but have left " << kTestTotalShards << " unset.\n";
+ const Message msg = Message() << "Invalid environment variables: you have "
+ << kTestShardIndex << " = " << shard_index
+ << ", but have left " << kTestTotalShards
+ << " unset.\n";
ColoredPrintf(GTestColor::kRed, "%s", msg.GetString().c_str());
fflush(stdout);
exit(EXIT_FAILURE);
} else if (total_shards != -1 && shard_index == -1) {
const Message msg = Message()
- << "Invalid environment variables: you have "
- << kTestTotalShards << " = " << total_shards
- << ", but have left " << kTestShardIndex << " unset.\n";
+ << "Invalid environment variables: you have "
+ << kTestTotalShards << " = " << total_shards
+ << ", but have left " << kTestShardIndex << " unset.\n";
ColoredPrintf(GTestColor::kRed, "%s", msg.GetString().c_str());
fflush(stdout);
exit(EXIT_FAILURE);
} else if (shard_index < 0 || shard_index >= total_shards) {
- const Message msg = Message()
- << "Invalid environment variables: we require 0 <= "
- << kTestShardIndex << " < " << kTestTotalShards
- << ", but you have " << kTestShardIndex << "=" << shard_index
- << ", " << kTestTotalShards << "=" << total_shards << ".\n";
+ const Message msg =
+ Message() << "Invalid environment variables: we require 0 <= "
+ << kTestShardIndex << " < " << kTestTotalShards
+ << ", but you have " << kTestShardIndex << "=" << shard_index
+ << ", " << kTestTotalShards << "=" << total_shards << ".\n";
ColoredPrintf(GTestColor::kRed, "%s", msg.GetString().c_str());
fflush(stdout);
exit(EXIT_FAILURE);
@@ -6022,11 +6040,16 @@
// https://github.com/google/googletest/blob/master/googletest/docs/advanced.md
// . Returns the number of tests that should run.
int UnitTestImpl::FilterTests(ReactionToSharding shard_tests) {
- const int32_t total_shards = shard_tests == HONOR_SHARDING_PROTOCOL ?
- Int32FromEnvOrDie(kTestTotalShards, -1) : -1;
- const int32_t shard_index = shard_tests == HONOR_SHARDING_PROTOCOL ?
- Int32FromEnvOrDie(kTestShardIndex, -1) : -1;
+ const int32_t total_shards = shard_tests == HONOR_SHARDING_PROTOCOL
+ ? Int32FromEnvOrDie(kTestTotalShards, -1)
+ : -1;
+ const int32_t shard_index = shard_tests == HONOR_SHARDING_PROTOCOL
+ ? Int32FromEnvOrDie(kTestShardIndex, -1)
+ : -1;
+ const PositiveAndNegativeUnitTestFilter gtest_flag_filter(
+ GTEST_FLAG_GET(filter));
+ const UnitTestFilter disable_test_filter(kDisableTestFilter);
// num_runnable_tests are the number of tests that will
// run across all shards (i.e., match filter and are not disabled).
// num_selected_tests are the number of tests to be run on
@@ -6042,18 +6065,17 @@
const std::string test_name(test_info->name());
// A test is disabled if test suite name or test name matches
// kDisableTestFilter.
- const bool is_disabled = internal::UnitTestOptions::MatchesFilter(
- test_suite_name, kDisableTestFilter) ||
- internal::UnitTestOptions::MatchesFilter(
- test_name, kDisableTestFilter);
+ const bool is_disabled =
+ disable_test_filter.MatchesName(test_suite_name) ||
+ disable_test_filter.MatchesName(test_name);
test_info->is_disabled_ = is_disabled;
- const bool matches_filter = internal::UnitTestOptions::FilterMatchesTest(
- test_suite_name, test_name);
+ const bool matches_filter =
+ gtest_flag_filter.MatchesTest(test_suite_name, test_name);
test_info->matches_filter_ = matches_filter;
const bool is_runnable =
- (GTEST_FLAG(also_run_disabled_tests) || !is_disabled) &&
+ (GTEST_FLAG_GET(also_run_disabled_tests) || !is_disabled) &&
matches_filter;
const bool is_in_another_shard =
@@ -6222,8 +6244,8 @@
// For example, if Foo() calls Bar(), which in turn calls
// GetCurrentOsStackTraceExceptTop(..., 1), Foo() will be included in
// the trace but Bar() and GetCurrentOsStackTraceExceptTop() won't.
-std::string GetCurrentOsStackTraceExceptTop(UnitTest* /*unit_test*/,
- int skip_count) {
+GTEST_NO_INLINE_ GTEST_NO_TAIL_CALL_ std::string
+GetCurrentOsStackTraceExceptTop(UnitTest* /*unit_test*/, int skip_count) {
// We pass skip_count + 1 to skip this wrapper function in addition
// to what the user really wants to skip.
return GetUnitTestImpl()->CurrentOsStackTraceExceptTop(skip_count + 1);
@@ -6233,7 +6255,7 @@
// suppress unreachable code warnings.
namespace {
class ClassUniqueToAlwaysTrue {};
-}
+} // namespace
bool IsTrue(bool condition) { return condition; }
@@ -6241,8 +6263,7 @@
#if GTEST_HAS_EXCEPTIONS
// This condition is always false so AlwaysTrue() never actually throws,
// but it makes the compiler think that it may throw.
- if (IsTrue(false))
- throw ClassUniqueToAlwaysTrue();
+ if (IsTrue(false)) throw ClassUniqueToAlwaysTrue();
#endif // GTEST_HAS_EXCEPTIONS
return true;
}
@@ -6264,13 +6285,14 @@
// part can be omitted.
//
// Returns the value of the flag, or NULL if the parsing failed.
-static const char* ParseFlagValue(const char* str, const char* flag,
+static const char* ParseFlagValue(const char* str, const char* flag_name,
bool def_optional) {
// str and flag must not be NULL.
- if (str == nullptr || flag == nullptr) return nullptr;
+ if (str == nullptr || flag_name == nullptr) return nullptr;
// The flag must start with "--" followed by GTEST_FLAG_PREFIX_.
- const std::string flag_str = std::string("--") + GTEST_FLAG_PREFIX_ + flag;
+ const std::string flag_str =
+ std::string("--") + GTEST_FLAG_PREFIX_ + flag_name;
const size_t flag_len = flag_str.length();
if (strncmp(str, flag_str.c_str(), flag_len) != 0) return nullptr;
@@ -6301,9 +6323,9 @@
//
// On success, stores the value of the flag in *value, and returns
// true. On failure, returns false without changing *value.
-static bool ParseBoolFlag(const char* str, const char* flag, bool* value) {
+static bool ParseFlag(const char* str, const char* flag_name, bool* value) {
// Gets the value of the flag as a string.
- const char* const value_str = ParseFlagValue(str, flag, true);
+ const char* const value_str = ParseFlagValue(str, flag_name, true);
// Aborts if the parsing failed.
if (value_str == nullptr) return false;
@@ -6317,16 +6339,16 @@
//
// On success, stores the value of the flag in *value, and returns
// true. On failure, returns false without changing *value.
-bool ParseInt32Flag(const char* str, const char* flag, int32_t* value) {
+bool ParseFlag(const char* str, const char* flag_name, int32_t* value) {
// Gets the value of the flag as a string.
- const char* const value_str = ParseFlagValue(str, flag, false);
+ const char* const value_str = ParseFlagValue(str, flag_name, false);
// Aborts if the parsing failed.
if (value_str == nullptr) return false;
// Sets *value to the value of the flag.
- return ParseInt32(Message() << "The value of flag --" << flag,
- value_str, value);
+ return ParseInt32(Message() << "The value of flag --" << flag_name, value_str,
+ value);
}
// Parses a string for a string flag, in the form of "--flag=value".
@@ -6334,9 +6356,9 @@
// On success, stores the value of the flag in *value, and returns
// true. On failure, returns false without changing *value.
template <typename String>
-static bool ParseStringFlag(const char* str, const char* flag, String* value) {
+static bool ParseFlag(const char* str, const char* flag_name, String* value) {
// Gets the value of the flag as a string.
- const char* const value_str = ParseFlagValue(str, flag, false);
+ const char* const value_str = ParseFlagValue(str, flag_name, false);
// Aborts if the parsing failed.
if (value_str == nullptr) return false;
@@ -6353,8 +6375,7 @@
// GTEST_INTERNAL_PREFIX_ followed by "internal_" are considered Google Test
// internal flags and do not trigger the help message.
static bool HasGoogleTestFlagPrefix(const char* str) {
- return (SkipPrefix("--", &str) ||
- SkipPrefix("-", &str) ||
+ return (SkipPrefix("--", &str) || SkipPrefix("-", &str) ||
SkipPrefix("/", &str)) &&
!SkipPrefix(GTEST_FLAG_PREFIX_ "internal_", &str) &&
(SkipPrefix(GTEST_FLAG_PREFIX_, &str) ||
@@ -6437,6 +6458,10 @@
"random_seed=@Y[NUMBER]@D\n"
" Random number seed to use for shuffling test orders (between 1 and\n"
" 99999, or 0 to use a seed based on the current time).\n"
+ " @G--" GTEST_FLAG_PREFIX_
+ "recreate_environments_when_repeating@D\n"
+ " Sets up and tears down the global test environment on each repeat\n"
+ " of the test.\n"
"\n"
"Test Output:\n"
" @G--" GTEST_FLAG_PREFIX_
@@ -6454,18 +6479,18 @@
" Generate a JSON or XML report in the given directory or with the "
"given\n"
" file name. @YFILE_PATH@D defaults to @Gtest_detail.xml@D.\n"
-# if GTEST_CAN_STREAM_RESULTS_
+#if GTEST_CAN_STREAM_RESULTS_
" @G--" GTEST_FLAG_PREFIX_
"stream_result_to=@YHOST@G:@YPORT@D\n"
" Stream test results to the given server.\n"
-# endif // GTEST_CAN_STREAM_RESULTS_
+#endif // GTEST_CAN_STREAM_RESULTS_
"\n"
"Assertion Behavior:\n"
-# if GTEST_HAS_DEATH_TEST && !GTEST_OS_WINDOWS
+#if GTEST_HAS_DEATH_TEST && !GTEST_OS_WINDOWS
" @G--" GTEST_FLAG_PREFIX_
"death_test_style=@Y(@Gfast@Y|@Gthreadsafe@Y)@D\n"
" Set the default death test style.\n"
-# endif // GTEST_HAS_DEATH_TEST && !GTEST_OS_WINDOWS
+#endif // GTEST_HAS_DEATH_TEST && !GTEST_OS_WINDOWS
" @G--" GTEST_FLAG_PREFIX_
"break_on_failure@D\n"
" Turn assertion failures into debugger break-points.\n"
@@ -6497,41 +6522,44 @@
"@G<" GTEST_DEV_EMAIL_ ">@D.\n";
static bool ParseGoogleTestFlag(const char* const arg) {
- return ParseBoolFlag(arg, kAlsoRunDisabledTestsFlag,
- >EST_FLAG(also_run_disabled_tests)) ||
- ParseBoolFlag(arg, kBreakOnFailureFlag,
- >EST_FLAG(break_on_failure)) ||
- ParseBoolFlag(arg, kCatchExceptionsFlag,
- >EST_FLAG(catch_exceptions)) ||
- ParseStringFlag(arg, kColorFlag, >EST_FLAG(color)) ||
- ParseStringFlag(arg, kDeathTestStyleFlag,
- >EST_FLAG(death_test_style)) ||
- ParseBoolFlag(arg, kDeathTestUseFork,
- >EST_FLAG(death_test_use_fork)) ||
- ParseBoolFlag(arg, kFailFast, >EST_FLAG(fail_fast)) ||
- ParseStringFlag(arg, kFilterFlag, >EST_FLAG(filter)) ||
- ParseStringFlag(arg, kInternalRunDeathTestFlag,
- >EST_FLAG(internal_run_death_test)) ||
- ParseBoolFlag(arg, kListTestsFlag, >EST_FLAG(list_tests)) ||
- ParseStringFlag(arg, kOutputFlag, >EST_FLAG(output)) ||
- ParseBoolFlag(arg, kBriefFlag, >EST_FLAG(brief)) ||
- ParseBoolFlag(arg, kPrintTimeFlag, >EST_FLAG(print_time)) ||
- ParseBoolFlag(arg, kPrintUTF8Flag, >EST_FLAG(print_utf8)) ||
- ParseInt32Flag(arg, kRandomSeedFlag, >EST_FLAG(random_seed)) ||
- ParseInt32Flag(arg, kRepeatFlag, >EST_FLAG(repeat)) ||
- ParseBoolFlag(arg, kShuffleFlag, >EST_FLAG(shuffle)) ||
- ParseInt32Flag(arg, kStackTraceDepthFlag,
- >EST_FLAG(stack_trace_depth)) ||
- ParseStringFlag(arg, kStreamResultToFlag,
- >EST_FLAG(stream_result_to)) ||
- ParseBoolFlag(arg, kThrowOnFailureFlag, >EST_FLAG(throw_on_failure));
+#define GTEST_INTERNAL_PARSE_FLAG(flag_name) \
+ do { \
+ auto value = GTEST_FLAG_GET(flag_name); \
+ if (ParseFlag(arg, #flag_name, &value)) { \
+ GTEST_FLAG_SET(flag_name, value); \
+ return true; \
+ } \
+ } while (false)
+
+ GTEST_INTERNAL_PARSE_FLAG(also_run_disabled_tests);
+ GTEST_INTERNAL_PARSE_FLAG(break_on_failure);
+ GTEST_INTERNAL_PARSE_FLAG(catch_exceptions);
+ GTEST_INTERNAL_PARSE_FLAG(color);
+ GTEST_INTERNAL_PARSE_FLAG(death_test_style);
+ GTEST_INTERNAL_PARSE_FLAG(death_test_use_fork);
+ GTEST_INTERNAL_PARSE_FLAG(fail_fast);
+ GTEST_INTERNAL_PARSE_FLAG(filter);
+ GTEST_INTERNAL_PARSE_FLAG(internal_run_death_test);
+ GTEST_INTERNAL_PARSE_FLAG(list_tests);
+ GTEST_INTERNAL_PARSE_FLAG(output);
+ GTEST_INTERNAL_PARSE_FLAG(brief);
+ GTEST_INTERNAL_PARSE_FLAG(print_time);
+ GTEST_INTERNAL_PARSE_FLAG(print_utf8);
+ GTEST_INTERNAL_PARSE_FLAG(random_seed);
+ GTEST_INTERNAL_PARSE_FLAG(repeat);
+ GTEST_INTERNAL_PARSE_FLAG(recreate_environments_when_repeating);
+ GTEST_INTERNAL_PARSE_FLAG(shuffle);
+ GTEST_INTERNAL_PARSE_FLAG(stack_trace_depth);
+ GTEST_INTERNAL_PARSE_FLAG(stream_result_to);
+ GTEST_INTERNAL_PARSE_FLAG(throw_on_failure);
+ return false;
}
#if GTEST_USE_OWN_FLAGFILE_FLAG_
static void LoadFlagsFromFile(const std::string& path) {
FILE* flagfile = posix::FOpen(path.c_str(), "r");
if (!flagfile) {
- GTEST_LOG_(FATAL) << "Unable to open file \"" << GTEST_FLAG(flagfile)
+ GTEST_LOG_(FATAL) << "Unable to open file \"" << GTEST_FLAG_GET(flagfile)
<< "\"";
}
std::string contents(ReadEntireFile(flagfile));
@@ -6539,10 +6567,8 @@
std::vector<std::string> lines;
SplitString(contents, '\n', &lines);
for (size_t i = 0; i < lines.size(); ++i) {
- if (lines[i].empty())
- continue;
- if (!ParseGoogleTestFlag(lines[i].c_str()))
- g_help_flag = true;
+ if (lines[i].empty()) continue;
+ if (!ParseGoogleTestFlag(lines[i].c_str())) g_help_flag = true;
}
}
#endif // GTEST_USE_OWN_FLAGFILE_FLAG_
@@ -6552,25 +6578,23 @@
// instantiated to either char or wchar_t.
template <typename CharType>
void ParseGoogleTestFlagsOnlyImpl(int* argc, CharType** argv) {
+ std::string flagfile_value;
for (int i = 1; i < *argc; i++) {
const std::string arg_string = StreamableToString(argv[i]);
const char* const arg = arg_string.c_str();
- using internal::ParseBoolFlag;
- using internal::ParseInt32Flag;
- using internal::ParseStringFlag;
+ using internal::ParseFlag;
bool remove_flag = false;
if (ParseGoogleTestFlag(arg)) {
remove_flag = true;
#if GTEST_USE_OWN_FLAGFILE_FLAG_
- } else if (ParseStringFlag(arg, kFlagfileFlag, >EST_FLAG(flagfile))) {
- LoadFlagsFromFile(GTEST_FLAG(flagfile));
+ } else if (ParseFlag(arg, "flagfile", &flagfile_value)) {
+ GTEST_FLAG_SET(flagfile, flagfile_value);
+ LoadFlagsFromFile(flagfile_value);
remove_flag = true;
#endif // GTEST_USE_OWN_FLAGFILE_FLAG_
- } else if (arg_string == "--help" || arg_string == "-h" ||
- arg_string == "-?" || arg_string == "/?" ||
- HasGoogleTestFlagPrefix(arg)) {
+ } else if (arg_string == "--help" || HasGoogleTestFlagPrefix(arg)) {
// Both help flag and unrecognized Google Test flags (excluding
// internal ones) trigger help display.
g_help_flag = true;
@@ -6605,7 +6629,27 @@
// Parses the command line for Google Test flags, without initializing
// other parts of Google Test.
void ParseGoogleTestFlagsOnly(int* argc, char** argv) {
+#if GTEST_HAS_ABSL
+ if (*argc > 0) {
+ // absl::ParseCommandLine() requires *argc > 0.
+ auto positional_args = absl::flags_internal::ParseCommandLineImpl(
+ *argc, argv, absl::flags_internal::ArgvListAction::kRemoveParsedArgs,
+ absl::flags_internal::UsageFlagsAction::kHandleUsage,
+ absl::flags_internal::OnUndefinedFlag::kReportUndefined);
+ // Any command-line positional arguments not part of any command-line flag
+ // (or arguments to a flag) are copied back out to argv, with the program
+ // invocation name at position 0, and argc is resized. This includes
+ // positional arguments after the flag-terminating delimiter '--'.
+ // See https://abseil.io/docs/cpp/guides/flags.
+ std::copy(positional_args.begin(), positional_args.end(), argv);
+ if (static_cast<int>(positional_args.size()) < *argc) {
+ argv[positional_args.size()] = nullptr;
+ *argc = static_cast<int>(positional_args.size());
+ }
+ }
+#else
ParseGoogleTestFlagsOnlyImpl(argc, argv);
+#endif
// Fix the value of *_NSGetArgc() on macOS, but if and only if
// *_NSGetArgv() == argv
@@ -6640,6 +6684,12 @@
#if GTEST_HAS_ABSL
absl::InitializeSymbolizer(g_argvs[0].c_str());
+
+ // When using the Abseil Flags library, set the program usage message to the
+ // help message, but remove the color-encoding from the message first.
+ absl::SetProgramUsageMessage(absl::StrReplaceAll(
+ kColorEncodedHelpMessage,
+ {{"@D", ""}, {"@R", ""}, {"@G", ""}, {"@Y", ""}, {"@@", "@"}}));
#endif // GTEST_HAS_ABSL
ParseGoogleTestFlagsOnly(argc, argv);
@@ -6660,7 +6710,7 @@
void InitGoogleTest(int* argc, char** argv) {
#if defined(GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_)
GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_(argc, argv);
-#else // defined(GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_)
+#else // defined(GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_)
internal::InitGoogleTestImpl(argc, argv);
#endif // defined(GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_)
}
@@ -6670,7 +6720,7 @@
void InitGoogleTest(int* argc, wchar_t** argv) {
#if defined(GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_)
GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_(argc, argv);
-#else // defined(GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_)
+#else // defined(GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_)
internal::InitGoogleTestImpl(argc, argv);
#endif // defined(GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_)
}
@@ -6686,42 +6736,42 @@
#if defined(GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_)
GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_(&argc, argv);
-#else // defined(GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_)
+#else // defined(GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_)
internal::InitGoogleTestImpl(&argc, argv);
#endif // defined(GTEST_CUSTOM_INIT_GOOGLE_TEST_FUNCTION_)
}
+#if !defined(GTEST_CUSTOM_TEMPDIR_FUNCTION_)
+// Return value of first environment variable that is set and contains
+// a non-empty string. If there are none, return the "fallback" string.
+// Since we like the temporary directory to have a directory separator suffix,
+// add it if not provided in the environment variable value.
+static std::string GetTempDirFromEnv(
+ std::initializer_list<const char*> environment_variables,
+ const char* fallback, char separator) {
+ for (const char* variable_name : environment_variables) {
+ const char* value = internal::posix::GetEnv(variable_name);
+ if (value != nullptr && value[0] != '\0') {
+ if (value[strlen(value) - 1] != separator) {
+ return std::string(value).append(1, separator);
+ }
+ return value;
+ }
+ }
+ return fallback;
+}
+#endif
+
std::string TempDir() {
#if defined(GTEST_CUSTOM_TEMPDIR_FUNCTION_)
return GTEST_CUSTOM_TEMPDIR_FUNCTION_();
-#elif GTEST_OS_WINDOWS_MOBILE
- return "\\temp\\";
-#elif GTEST_OS_WINDOWS
- const char* temp_dir = internal::posix::GetEnv("TEMP");
- if (temp_dir == nullptr || temp_dir[0] == '\0') {
- return "\\temp\\";
- } else if (temp_dir[strlen(temp_dir) - 1] == '\\') {
- return temp_dir;
- } else {
- return std::string(temp_dir) + "\\";
- }
+#elif GTEST_OS_WINDOWS || GTEST_OS_WINDOWS_MOBILE
+ return GetTempDirFromEnv({"TEST_TMPDIR", "TEMP"}, "\\temp\\", '\\');
#elif GTEST_OS_LINUX_ANDROID
- const char* temp_dir = internal::posix::GetEnv("TEST_TMPDIR");
- if (temp_dir == nullptr || temp_dir[0] == '\0') {
- return "/data/local/tmp/";
- } else {
- return temp_dir;
- }
-#elif GTEST_OS_LINUX
- const char* temp_dir = internal::posix::GetEnv("TEST_TMPDIR");
- if (temp_dir == nullptr || temp_dir[0] == '\0') {
- return "/tmp/";
- } else {
- return temp_dir;
- }
+ return GetTempDirFromEnv({"TEST_TMPDIR", "TMPDIR"}, "/data/local/tmp/", '/');
#else
- return "/tmp/";
-#endif // GTEST_OS_WINDOWS_MOBILE
+ return GetTempDirFromEnv({"TEST_TMPDIR", "TMPDIR"}, "/tmp/", '/');
+#endif
}
// Class ScopedTrace
@@ -6738,8 +6788,7 @@
}
// Pops the info pushed by the c'tor.
-ScopedTrace::~ScopedTrace()
- GTEST_LOCK_EXCLUDED_(&UnitTest::mutex_) {
+ScopedTrace::~ScopedTrace() GTEST_LOCK_EXCLUDED_(&UnitTest::mutex_) {
UnitTest::GetInstance()->PopGTestTrace();
}
diff --git a/third_party/googletest/src/googletest/src/gtest_main.cc b/third_party/googletest/src/googletest/src/gtest_main.cc
index 46b27c3..4497637 100644
--- a/third_party/googletest/src/googletest/src/gtest_main.cc
+++ b/third_party/googletest/src/googletest/src/gtest_main.cc
@@ -28,15 +28,14 @@
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#include <cstdio>
+
#include "gtest/gtest.h"
#if GTEST_OS_ESP8266 || GTEST_OS_ESP32
#if GTEST_OS_ESP8266
extern "C" {
#endif
-void setup() {
- testing::InitGoogleTest();
-}
+void setup() { testing::InitGoogleTest(); }
void loop() { RUN_ALL_TESTS(); }
