blob: 22e1314ec2964cd6f9bed3983f5a2cc30f4ddf3b [file] [log] [blame]
This testing framework is initial developed by Intel for convex hull study.
It is contributed to Alliance of Open Media for convex hull study and future AV2
Common Test Condition (CTC) testing.
For questions and technical support, please contact ryan.lei@intel.com or
maggie.sun@intel.com
This readme file provides a simple introduction of the framework and steps to
use it for different studies.
Prerequisites
1. Inside ./bin folder, make sure you have executables for all external tools,
such as vmafossexec, ffmpeg, HDRMetrics, encoder, decoder, etc. You can get
the pre-build binary or source code for the following tools that are needed:
ffmpeg: https://ffmpeg.org/download.html
vmaf tools: https://github.com/Netflix/vmaf
HDRMetrics: https://gitlab.com/standards/HDRTools
libaom AV1 encoder/decoder: https://aomedia.googlesource.com/aom/
SVT HEVC encoder: https://github.com/OpenVisualCloud/SVT-HEVC
SVT AV1 encoder: https://github.com/OpenVisualCloud/SVT-AV1
Please always follow the manual of these tools to update the command
parameters used in the script
2. VMAF requires a model file to run. currently, vmaf_v0.6.1 model files
(vmaf_v0.6.1.pkl.model, vmaf_v0.6.1.pkl) are used and stored under the ./bin
folder.
3. HDRMetrics requires a template config file (HDRMetricsYUV420_template.cfg).
It is also located under the ./bin folder. Test framework will generate
individual config file based on this template.
4. VBA macro binary file (vbaProject-AV2.bin) for bdrate calculation is also
stored under ./bin folder, which is needed for BDRATE calculation inside the
excel file.
5. Test framework is implemented in python 3. It requires few extra python
packages, such as xlrd, xlsxwriter, argparse, etc.
Things you need to update inside ./src/Config.py to configure the test.
1. Update the Clips table following the existing example to provide the list of
video sequences that you want to test. The first value is the yuv file name
without .yuv suffix. All test video sequences should be in the folder
specified by ContentPath. Test framework will extract the string before the
first "_" as the short name to identify the test sequence and use that to
generate file name of other intermediate files, so please make sure the short
name of the test sequence is unique. Currently only 8 bit yuv sequences in
yuv420p format are supported.
2. Update FrameNum to specify number of frames you want to process.
3. Update DnScaleRatio list to provide the list of downscaling ratios.
4. Update DnScalingAlgos and UpScalingAlgos list to specify downscaling/upscaling
filter types that you want to test. The name of the filter types is that
supported by ffmpeg. The size of these two lists must be the same. Filter
types for downscaling and upscaling can be different.
5. Update QPs list to specify the list of QPs you want to test for different
codecs. QPs must be in the valid QP range for different coding standards.
For example, [0, 51] for hevc and [0, 63] for AV1
6. Update QualityEvalMethods to specify tools you want to use to calculate
quality metrics in the QualityList. Currently, only VMAF and HDRTools are
supported. You also need to update QualityList and QualityEvalMethods to
specify what quality metrics you want to calculate. Test framework will filter
out other metrics and only report the metrics that are specified.
7. Update LogCmdOnly flag. When it is set to True, the test framework will only
capture all process command sequences into a log file (under ./test/testLogs
folder with time stamp) without actually running it on your system. This
feature is to support the use case in which actual processing tasks need to
distributed on to a server cluster and executed in parallel.
8. Update SMOKE_TEST flag, It is used for sanity check purpose, in which only
few frames are processed to reduce the test time.
Command lines for running the tests:
below is the full command line options in help message:
-h, --help show this help message and exit
-f, --function function to run: clean, scaling, sumscaling,
encode, convexhull, summary
-k, --KeepUpscaleOutput [0|1] in function clean, if keep upscaled yuv
files. It is false by default
-s, --SaveMemory [0|1] save memory mode will delete most files in
intermediate steps and keeps only necessary ones
for RD calculation. It is false by default
-l, --LoggingLevel logging level: 0:No Logging, 1: Critical, 2: Error,
3:Warning, 4: Info, 5: Debug
-c, --CodecName CodecName: av1 or hevc
-m, --EncodeMethod EncodeMethod: ffmpeg, aom, svt
-p, --EncodePreset EncodePreset: medium, slow, fast, etc for ffmpeg,
0,1,2... for aom and svt
Sample command for typical operations:
1. python ConvexHullTest.py -f clean
This command will clean up all intermediate files under ./test folder
2. python ConvexHullTest.py -f scaling
This command will run the standalone downscaling and upscaling tests.
Downscaled YUV files are stored under
./test/downscaledYUVs folder. Upscaled YUV files are stored under
./test/upscaledYUVs folder. Quality metrics log files are stored under
./test/qualityLogs folder. Other processing logs and command logs are stored
under ./test/testLogs folder. In case HDRMetric is used, individual config
files are generated and stored under ./test/configFiles folder. All
intermediate file names indicate the input, output resolution and filter types
that are used.
3. python ConvexHullTest.py -f sumscaling
This command will summarize the quality metrics for the scaling test into
excel files under ./analysis/scalingresult folder. There are excel files for
each individual test sequence and also excel file that summarizes quality
result for all test sequences based on classes.
4. python ConvexHullTest.py -f encode -c hevc -m ffmpeg -p medium
This command will run the encoding test. It actually contains downscale
(optional), encode, decode, upscale, quality metrics steps. Different
encoding method and codec name can be specified. For a particular encoder,
encoding preset should also be specified. Downscale step will be skipped if
downscaled yuv files already generated in ./test/downscaledYUVs. Encoded
bitstreams are stored under ./test/bitstreams folder. Decoded YUV files are
stored under ./test/decodedYUVs folder. Decoded and then upscaled YUV files
are stored under ./test/decUpscaledYUVs folder. Quality logs are stored under
./test/qualityLogs folder.
5. python ConvexHullTest.py -f convexhull -c hevc -m ffmpeg -p medium
This command will summarize the per sequence quality result based on
different scaling ratios and scaling filter types.
Please make sure the same encoding method/codec name/preset are used as the
previous steps. Output excel files are stored under ./analysis/rdresult
folder. For each sequence and downscaling/upscaling filter type, an excel
sheet is generated that contains the bitrate and quality metric for different
downscaled resolutions. Rate distortion curve for all quality metrics will
be drawn in a scatter plot. Convex hull will be calculated and draw on top
of the rate distortion curve.
In this step, a summary excel file is also generated, which include the
convex hull rate distortion points for all test content and downscaling and
upscaling algorithm pair. This excel file is needed when you want to
calculate bdrate between two encoding runs to evaluate the quality impact on
overall convex hull from a coding tool.
6. python ConvexHullTest.py -f summary -c hevc -m ffmpeg -p medium
This command will summarize the quality metrics across all test sequences
into an excel file stored under the ./analysis/summary folder. BDRATE between
different resolutions will be calculated. Average result based on content
classes will be calculated.
In order to calculate BDRATE between convex hull from two encoding test runs
with different codec or different encoding preset, ./src/ConvexHullBDRate.py
script is needed. Command line options for this script is as following:
usage: ConvexHullBDRate.py [options]
optional arguments:
-h, --help show this help message and exit
-i1 , --input1 convex hull summary excel file for base mode
-i2 , --input2 convex hull summary excel file for target mode
-o , --output output excel file with BDRATE for base and target modes
To use this script, the convex hull summary excel files for base and target
modes are needed. Script will parse the rate distortion data from the convex hull
for each individual test content, then it will use the bdrate macro to calculate
the bdrate between two convex hulls.
a sample command is:
python ConvexHullBDRate.py -i1 ConvexHullRD_ffmpeg_hevc_medium.xlsx
-i2 ConvexHullRD_ffmpeg_hevc_veryslow.xlsx -o ConvexHullBDRate.xlsm