OpenVINO 2022.3之九：Post-training Optimization Tool (POT)
Post-training Optimization Tool (POT) 通过在已训练好的模型上应用量化算法，将模型的权重和激活函数从 FP32/FP16 的值域映射到 INT8 的值域中，从而实现模型压缩，以降低模型推理所需的计算资源和内存带宽，进一步提高模型的推理性能。不同于 Quantization-aware Training(QAT) 方法，POT在不需要对原模型进行 fine-tuning 的情况下进行量化，也能得到精度较好的 INT8 模型，因此广泛地被应用于工业界的量化实践中。
POT提供了两种量化算法: Default Quantization 和 Accuracy-aware Quantization，
-
Default Quantization (DQ) 提供了一种快速的量化方法，量化后的模型在大多数情况下能够提供较好的精度，适合作为模型 INT8 量化的 baseline。
{ "name": "DefaultQuantization", # Optimization algorithm name "params": { "preset": "performance", # Preset [performance, mixed] which controls # the quantization scheme. For the CPU: # performance - symmetric quantization of weights and activations. # mixed - symmetric weights and asymmetric activations. "stat_subset_size": 300 # Size of subset to calculate activations statistics that can be used # for quantization parameters calculation. } } -
Accuracy-aware Quantization ( AAQ ) 是一种基于 Default Quantization 上的迭代量化算法，以 DQ 量化后的模型作为 baseline，若 INT8 模型精度达到预期精度范围，则停止迭代，反之，量化算法会分析模型各层对精度的影响，并将对精度影响最大的层回退到FP32精度，然后重新评估模型精度，重复以上流程，直至模型达到预期精度范围。
{ "name": "DefaultQuantization", "params": { "preset": "performance", "stat_subset_size": 300 "activations": { # defines activation "range_estimator": { # defines how to estimate statistics "max": { # right border of the quantizating floating-point range "aggregator": "max", # use max(x) to aggregate statistics over calibration dataset "type": "abs_max" # use abs(max(x)) to get per-sample statistics } } } } }
1 Default Quantization (DQ)
提供了 POT 量化流水线通用化的接口，包括 DataLoader 和 Metric 等基类，用户可以通过继承 DataLoader 来定义客制化的数据集加载及预处理模块，通过继承 Metric 来定义客制化的后处理和精度计算的模块。这种方式更加灵活，可以适用不同客制化模型的量化需求.
1-1 Prepare data and dataset interface
import os
import numpy as np
import cv2 as cv
from openvino.tools.pot import DataLoader
class ImageLoader(DataLoader):
""" Loads images from a folder """
def __init__(self, dataset_path):
# Use OpenCV to gather image files
# Collect names of image files
self._files = []
all_files_in_dir = os.listdir(dataset_path)
for name in all_files_in_dir:
file = os.path.join(dataset_path, name)
if cv.haveImageReader(file):
self._files.append(file)
# Define shape of the model
self._shape = (224,224)
def __len__(self):
""" Returns the length of the dataset """
return len(self._files)
def __getitem__(self, index):
""" Returns image data by index in the NCHW layout
Note: model-specific preprocessing is omitted, consider adding it here
"""
if index >= len(self):
raise IndexError("Index out of dataset size")
image = cv.imread(self._files[index]) # read image with OpenCV
image = cv.resize(image, self._shape) # resize to a target input size
image = np.expand_dims(image, 0) # add batch dimension
image = image.transpose(0, 3, 1, 2) # convert to NCHW layout
return image, None # annotation is set to None
1-2 Select quantization parameters
{
"name": "DefaultQuantization",
"params": {
"target_device": "ANY",
"stat_subset_size": 300,
"stat_batch_size": 1
},
}
1-3 Define and run quantization process
from openvino.tools.pot import IEEngine
from openvino.tools.pot load_model, save_model
from openvino.tools.pot import compress_model_weights
from openvino.tools.pot import create_pipeline
# Model config specifies the name of the model and paths to .xml and .bin files of the model.
model_config =
{
"model_name": "model",
"model": path_to_xml,
"weights": path_to_bin,
}
# Engine config.
engine_config = {"device": "CPU"}
algorithms = [
{
"name": "DefaultQuantization",
"params": {
"target_device": "ANY",
"stat_subset_size": 300,
"stat_batch_size": 1
},
}
]
# Step 1: Implement and create a user data loader.
data_loader = ImageLoader("<path_to_images>")
# Step 2: Load a model.
model = load_model(model_config=model_config)
# Step 3: Initialize the engine for metric calculation and statistics collection.
engine = IEEngine(config=engine_config, data_loader=data_loader)
# Step 4: Create a pipeline of compression algorithms and run it.
pipeline = create_pipeline(algorithms, engine)
compressed_model = pipeline.run(model=model)
# Step 5 (Optional): Compress model weights to quantized precision
# to reduce the size of the final .bin file.
compress_model_weights(compressed_model)
# Step 6: Save the compressed model to the desired path.
# Set save_path to the directory where the model should be saved.
compressed_model_paths = save_model(
model=compressed_model,
save_path="optimized_model",
model_name="optimized_model",
)
2 Accuracy-aware Quantization ( AAQ )
2-1 Prepare data and dataset interface
同 2-1-1.
2-2 Define accuracy metric
from openvino.tools.pot import Metric
class Accuracy(Metric):
# Required methods
def __init__(self, top_k=1):
super().__init__()
self._top_k = top_k
self._name = 'accuracy@top{}'.format(self._top_k)
self._matches = [] # container of the results
@property
def value(self):
""" Returns accuracy metric value for all model outputs. """
return {self._name: self._matches[-1]}
@property
def avg_value(self):
""" Returns accuracy metric value for all model outputs. """
return {self._name: np.ravel(self._matches).mean()}
def update(self, output, target):
""" Updates prediction matches.
:param output: model output
:param target: annotations
"""
if len(output) > 1:
raise Exception('The accuracy metric cannot be calculated '
'for a model with multiple outputs')
if isinstance(target, dict):
target = list(target.values())
predictions = np.argsort(output[0], axis=1)[:, -self._top_k:]
match = [float(t in predictions[i]) for i, t in enumerate(target)]
self._matches.append(match)
def reset(self):
""" Resets collected matches """
self._matches = []
def get_attributes(self):
"""
Returns a dictionary of metric attributes {metric_name: {attribute_name: value}}.
Required attributes: 'direction': 'higher-better' or 'higher-worse'
'type': metric type
"""
return {self._name: {'direction': 'higher-better',
'type': 'accuracy'}}
调用:
metric = Accuracy()
engine = IEEngine(config=engine_config, data_loader=data_loader, metric=metric)
2-3 Select quantization parameters
Accuracy-aware Quantization所独有的唯一参数是 maximal_drop, 表明模型量化后必须实现的最大降低精度，默认是0.01(1%)
2-4 Define and run quantization process
from openvino.tools.pot import IEEngine
from openvino.tools.pot load_model, save_model
from openvino.tools.pot import compress_model_weights
from openvino.tools.pot import create_pipeline
# Model config specifies the model name and paths to model .xml and .bin file
model_config = Dict(
{
"model_name": "model",
"model": path_to_xml,
"weights": path_to_bin,
}
)
# Engine config
engine_config = Dict({"device": "CPU"})
algorithms = [
{
"name": "AccuracyAwareQuantization",
"params": {
"target_device": "ANY",
"stat_subset_size": 300,
'maximal_drop': 0.02
},
}
]
# Step 1: Implement and create user's data loader.
data_loader = UserDataLoader()
# Step 2: Implement and create user's data loader.
metric = Accuracy()
# Step 3: Load the model.
model = load_model(model_config=model_config)
# Step 4: Initialize the engine for metric calculation and statistics collection.
engine = IEEngine(config=engine_config, data_loader=data_loader, metric=metric)
# Step 5: Create a pipeline of compression algorithms and run it.
pipeline = create_pipeline(algorithms, engine)
compressed_model = pipeline.run(model=model)
# Step 6 (Optional): Compress model weights to quantized precision
# in order to reduce the size of the final .bin file.
compress_model_weights(compressed_model)
# Step 7: Save the compressed model to the desired path.
# Set save_path to the directory where the model should be saved.
compressed_model_paths = save_model(
model=compressed_model,
save_path="optimized_model",
model_name="optimized_model",
)
# Step 8 (Optional): Evaluate the compressed model. Print the results.
metric_results = pipeline.evaluate(compressed_model)
