blob: ab401156510870a72c6d768f4296f873f4017d50 [file] [log] [blame]
This testing framework is initial developed by Intel for convex hull study.
It is contributed to Alliance of Open Media for convex hull study and future AV2
Common Test Condition (CTC) testing.
For questions and technical support, please contact ryan.lei@intel.com or
maggie.sun@intel.com
This readme file provides a simple introduction of the framework and steps to
use it for different studies.
Prerequisites
1. Inside ./bin folder, make sure you have executables for all external tools,
such as vmaf(version v2.1.1+), HDRConvert, encoder, decoder, etc. You can get
the pre-build binaries or source code for the following tools that are needed:
vmaf tools: https://github.com/Netflix/vmaf/releases/tag/v2.1.1
HDRConvert: https://gitlab.com/standards/HDRTools 0.20-dev branch
libaom research branch encoder/decoder: https://aomedia.googlesource.com/aom/
SVT AV1 encoder: https://github.com/OpenVisualCloud/SVT-AV1
Please always follow the manual of these tools to update the command
parameters used in the script.
when EnableTimingInfo is enabled, ptime utility is used on Windows to capture
the run time information for encode and decode. On Linux, time utility from
Linux system is used to capture run time information.
2. AV2 CTC has adopted VMAF tool as the reference implementation of all quality
metrics required. Using the versioned '--aom_ctc v1.0' at the command line
will generate all quality metrics for AV2 CTC.
3. HDRConvert is used to perform downscale and upscale operation. It requires
a template config file (HDRConvScalerY4MFile.cfg). It is also located under
the ./bin folder. Test framework will generate individual config file based
on this template.
4. VBA macro binary file (vbaProject-AV2.bin) for bdrate calculation is also
stored under ./bin folder, which is needed for BDRATE calculation inside the
excel file.
5. Test framework is implemented in python 3. It requires few extra python
packages, such as xlrd, xlsxwriter, argparse, etc.
Things you need to update to configure the test.
1. Update the test clips table in AV2CTCVideo.py file following the existing
example to provide the list and classes of video sequences that you want to
test. Currently, only .y4m files are supported. Test script will parse the
y4m file header to get the basic information of the video clip, such as
resolution, frame rate, bit depth and color format, etc.
2. Update the test configuration list in ./src/Config.py to specify the test
configurations. Currently, supported test configurations include Random Access
(RA), Low Delay (LD), All Intra(AI) and Adaptive Streaming (AS).
3. Update FrameNum in ./src/Config.py to specify number of frames you want to
process for different test configurations.
4. Update DnScaleRatio list in ./src/Config.py to provide the list of downscaling
ratios used for adaptive streaming (convex hull) test configuration.
5. Update DnScalingAlgos and UpScalingAlgos list in ./src/Config.py to specify
downscaling/upscaling filter types that you want to test. The name of the
filter types is that supported by HDRConvert. The size of these two lists
must be the same. Filter types for downscaling and upscaling can be different.
Right now, only the lanczos filter is supported.
6. Update QPs list in ./src/Config.py to specify the list of QPs you want to
test for different test configurations. QPs must be in the valid QP range for
different coding standards. For example, [0, 63] for AV1.
7. Update QualityList in ./src/Config.py to specify quality metrics you want to
calculate. Currently, only VMAF is supported for quality metrics calculation.
8. Update SMOKE_TEST flag, It is used for sanity check purpose, in which only
few frames are processed to reduce the test time.
Sample command lines for running the adaptive streaming (convex hull) test:
"python ConvexHullTest.py [options]"
below is the full command line options in help message:
-h, --help show this help message and exit
-f, --function function to run: clean, scaling, sumscaling,
encode, convexhull, summary
-k, --KeepUpscaleOutput [0|1] in function clean, if keep upscaled yuv
files. It is false by default
-s, --SaveMemory [0|1] save memory mode will delete most files in
intermediate steps and keeps only necessary ones
for RD calculation. It is false by default
-CmdOnly, --LogCmdOnly LogCmdOnly mode will only capture the command
sequences in the command log file instead of
actually run it. It is false by default
-l, --LoggingLevel logging level: 0:No Logging, 1: Critical, 2: Error,
3:Warning, 4: Info, 5: Debug
-c, --CodecName CodecName: av1
-m, --EncodeMethod EncodeMethod: aom, svt
-p, --EncodePreset EncodePreset: 0,1,2... for aom and svt
when LogCmdOnly is set to True, the test framework will only capture all process
command sequences into a log file (under ./test/testLogs folder with time stamp)
without actually running it on your system. This feature is to support the use
case in which actual processing tasks need to distributed on to a server cluster
and executed in parallel.
Sample command for typical operations:
1. python ConvexHullTest.py -f clean
This command will clean up all intermediate files under ./test folder
2. python ConvexHullTest.py -f scaling
This command will run the standalone downscaling and upscaling tests.
Downscaled YUV files are stored under ./test/downscaledYUVs folder.
Upscaled YUV files are stored under ./test/upscaledYUVs folder.
Quality metrics log files are stored under ./test/qualityLogs folder.
Other processing logs and command logs are stored under ./test/testLogs folder.
For using HDRConvert, individual config files are generated and stored under
./test/configFiles folder. All intermediate file names indicate the input,
output resolution and filter types that are used.
3. python ConvexHullTest.py -f sumscaling
This command will summarize the quality metrics for the scaling test into
excel files under ./analysis/scalingresult folder. There are excel files for
each individual test sequence and also excel file that summarizes quality
result for all test sequences based on classes.
4. python ConvexHullTest.py -f encode -c av1 -m aom -p 1
This command will run the encoding test. It actually contains downscale
(optional), encode, decode, upscale, quality metrics steps. Right now, only
av1 encoding/decoding with libaom is supported. Downscale step will be skipped
if downscaled yuv files already generated in ./test/downscaledYUVs. Encoded
bitstreams are stored under ./test/bitstreams folder. Decoded YUV files are
stored under ./test/decodedYUVs folder. Decoded and then upscaled YUV files
are stored under ./test/decUpscaledYUVs folder. Quality logs are stored under
./test/qualityLogs folder.
5. python ConvexHullTest.py -f convexhull -c av1 -m aom -p 1
This command will summarize the per sequence quality result based on
different scaling ratios and scaling filter types.
Please make sure the same encoding method/codec name/preset are used as the
previous steps. Output excel files are stored under ./analysis/rdresult
folder. For each sequence and downscaling/upscaling filter type, an excel
sheet is generated that contains the bitrate and quality metric for different
downscaled resolutions. Rate distortion curve for all quality metrics will
be drawn in a scatter plot. Convex hull will be calculated and draw on top
of the rate distortion curve.
In this step, a summary excel file is also generated, which include the
convex hull rate distortion points for all test content and downscaling and
upscaling algorithm pair. This excel file is needed when you want to
calculate bdrate between two encoding runs to evaluate the quality impact on
overall convex hull from a coding tool.
6. python ConvexHullTest.py -f summary -c av1 -m aom -p 1
This command will summarize the quality metrics across all test sequences
into an excel file stored under the ./analysis/summary folder. BDRATE between
different resolutions will be calculated. Average result based on content
classes will be calculated.
In order to calculate BDRATE between convex hull from two encoding test runs
with different codec or different encoding preset, ./src/ConvexHullBDRate.py
script is needed. Command line options for this script is as following:
usage: ConvexHullBDRate.py [options]
optional arguments:
-h, --help show this help message and exit
-i1 , --input1 convex hull summary excel file for base mode
-i2 , --input2 convex hull summary excel file for target mode
-o , --output output excel file with BDRATE for base and target modes
To use this script, the convex hull summary excel files for base and target
modes are needed. Script will parse the rate distortion data from the convex hull
for each individual test content, then it will use the bdrate macro to calculate
the bdrate between two convex hulls.
a sample command is:
python ConvexHullBDRate.py -i1 ConvexHullRD_ffmpeg_hevc_medium.xlsx
-i2 ConvexHullRD_ffmpeg_hevc_veryslow.xlsx -o ConvexHullBDRate.xlsm
Sample command lines for running the regular AV2 CTC test for All Intra(AI),
Low Delay(LD) and Random Access (RA) configurations:
"python AV2CTCTest.py [options]"
below is the full command line options in help message:
-h, --help show this help message and exit
-f, --function function to run: clean, encode, summary
-s, --SaveMemory [0|1] save memory mode will delete most files in
intermediate steps and keeps only necessary ones
for RD calculation. It is false by default
-CmdOnly, --LogCmdOnly LogCmdOnly mode will only capture the command
sequences in the command log file instead of
actually run it. It is false by default
-l, --LoggingLevel logging level: 0:No Logging, 1: Critical, 2: Error,
3:Warning, 4: Info, 5: Debug
-p, --EncodePreset EncodePreset: 0,1,2... for aom
Sample command for typical operations:
1. python AV2CTCTest.py -f clean
This command will clean up all intermediate files under ./test folder
2. python AV2CTCTest.py -f encode -p 1
This command will run the encoding test. It actually contains encode, decode,
and quality metrics steps. Right now, only av1 encoding/decoding with libaom
is supported. Encoded bitstreams are stored under ./test/bitstreams folder.
Decoded YUV files are stored under ./test/decodedYUVs folder. Quality logs
are stored under ./test/qualityLogs folder. Timing information logs are stored
under ./test/perfLogs floder.
3. python AV2CTCTest.py -f summary -p 1
This command will summarize the quality metrics across all test sequences
into an csv file stored under the ./analysis/summary folder.
4. after getting the csv file for two tests (reference and target), content of
the csv files can be directly copied into the excel template
(AV2Template_Vx.xlsm under the bin folder), which can calculate the BDRATE
of the two tests.