1. 引言
在人工智能迅速发展的今天，大型语言模型（LLMs）在多个领域展现出了巨大的潜力和应用价值。然而，如何评价这些模型的性能，了解它们的优缺点，成为了一个重要课题。OpenCompass，一个由上海人工智能实验室开发的大模型开源评测体系，提供了一套全面、公正、可复现的评测方案，帮助研究人员和开发者深入了解和优化他们的模型。
2. OpenCompass 简介
2.1 特点
- 开源可复现：确保评测过程的透明度和可重复性。
- 全面的能力维度：涵盖五大能力维度，使用70+数据集，约40万题目。
- 丰富的模型支持：支持20+ HuggingFace及API模型。
- 分布式高效评测：简化任务分割和分布式评测过程。
- 多样化评测范式：支持多种评测方式，包括零样本、小样本评测。
- 灵活化拓展：易于添加新模型、数据集或自定义任务分割策略。
2.2 评测对象
- 基座模型：强大的文本续写能力。
- 对话模型：优化的对话能力，理解人类指令
3. 评测操作
3.1 环境配置
- 创建开发机和conda环境。
- 面向GPU的环境搭建：安装依赖，包括Python、PyTorch、Transformers等。
- 拉取opencompass文件
-
studio-conda -o internlm-base -t opencompass source activate opencompass git clone -b 0.2.4 https://github.com/open-compass/opencompass cd opencompass pip install -e .如果pip install -e .安装未成功,请运行:
-
pip install -r requirements.txt
3.2 数据准备
- 下载并解压数据集至指定目录。
-
cp /share/temp/datasets/OpenCompassData-core-20231110.zip /root/opencompass/ unzip OpenCompassData-core-20231110.zip将会在 OpenCompass 下看到data文件夹
查看支持的数据集和模型
-
python tools/list_configs.py internlm ceval列出所有跟 InternLM 及 C-Eval 相关的配置
-
3.3 启动评测 (10% A100 8GB 资源)
- 使用命令行工具启动评测过程，监控输出结果。
命令行参数
--datasets：指定评测数据集。--hf-path：指定HuggingFace模型路径。--max-seq-len：设置最大序列长度。--batch-size：设置批量大小。--num-gpus：设置使用的GPU数量。--debug：开启调试模式。
确保按照上述步骤正确安装 OpenCompass 并准备好数据集后，可以通过以下命令评测 InternLM2-Chat-1.8B 模型在 C-Eval 数据集上的性能。由于 OpenCompass 默认并行启动评估过程，我们可以在第一次运行时以 --debug 模式启动评估，并检查是否存在问题。在 --debug 模式下，任务将按顺序执行，并实时打印输出。
python run.py --datasets ceval_gen --hf-path /share/new_models/Shanghai_AI_Laboratory/internlm2-chat-1_8b --tokenizer-path /share/new_models/Shanghai_AI_Laboratory/internlm2-chat-1_8b --tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True --model-kwargs trust_remote_code=True device_map='auto' --max-seq-len 1024 --max-out-len 16 --batch-size 2 --num-gpus 1 --debug遇到错误：
解决方案：
pip install protobuf
遇到错误mkl-service + Intel(R) MKL MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1
解决方案：
export MKL_SERVICE_FORCE_INTEL=1
#或
export MKL_THREADING_LAYER=GNU如果一切正常，您应该看到屏幕上显示 “Starting inference process”：
评测完成后，将会看到：
dataset version metric mode opencompass.models.huggingface.HuggingFace_Shanghai_AI_Laboratory_internlm2-chat-1_8b
---------------------------------------------- --------- ------------- ------ ---------------------------------------------------------------------------------------
ceval-computer_network db9ce2 accuracy gen 47.37
ceval-operating_system 1c2571 accuracy gen 47.37
ceval-computer_architecture a74dad accuracy gen 23.81
ceval-college_programming 4ca32a accuracy gen 13.51
ceval-college_physics 963fa8 accuracy gen 42.11
ceval-college_chemistry e78857 accuracy gen 33.33
ceval-advanced_mathematics ce03e2 accuracy gen 10.53
ceval-probability_and_statistics 65e812 accuracy gen 38.89
ceval-discrete_mathematics e894ae accuracy gen 25
ceval-electrical_engineer ae42b9 accuracy gen 27.03
ceval-metrology_engineer ee34ea accuracy gen 54.17
ceval-high_school_mathematics 1dc5bf accuracy gen 16.67
ceval-high_school_physics adf25f accuracy gen 42.11
ceval-high_school_chemistry 2ed27f accuracy gen 47.37
ceval-high_school_biology 8e2b9a accuracy gen 26.32
ceval-middle_school_mathematics bee8d5 accuracy gen 36.84
ceval-middle_school_biology 86817c accuracy gen 80.95
ceval-middle_school_physics 8accf6 accuracy gen 47.37
ceval-middle_school_chemistry 167a15 accuracy gen 80
ceval-veterinary_medicine b4e08d accuracy gen 43.48
ceval-college_economics f3f4e6 accuracy gen 32.73
ceval-business_administration c1614e accuracy gen 36.36
ceval-marxism cf874c accuracy gen 68.42
ceval-mao_zedong_thought 51c7a4 accuracy gen 70.83
ceval-education_science 591fee accuracy gen 55.17
ceval-teacher_qualification 4e4ced accuracy gen 59.09
ceval-high_school_politics 5c0de2 accuracy gen 57.89
ceval-high_school_geography 865461 accuracy gen 47.37
ceval-middle_school_politics 5be3e7 accuracy gen 71.43
ceval-middle_school_geography 8a63be accuracy gen 75
ceval-modern_chinese_history fc01af accuracy gen 52.17
ceval-ideological_and_moral_cultivation a2aa4a accuracy gen 73.68
ceval-logic f5b022 accuracy gen 27.27
ceval-law a110a1 accuracy gen 29.17
ceval-chinese_language_and_literature 0f8b68 accuracy gen 47.83
ceval-art_studies 2a1300 accuracy gen 42.42
ceval-professional_tour_guide 4e673e accuracy gen 51.72
ceval-legal_professional ce8787 accuracy gen 34.78
ceval-high_school_chinese 315705 accuracy gen 42.11
ceval-high_school_history 7eb30a accuracy gen 65
ceval-middle_school_history 48ab4a accuracy gen 86.36
ceval-civil_servant 87d061 accuracy gen 42.55
ceval-sports_science 70f27b accuracy gen 52.63
ceval-plant_protection 8941f9 accuracy gen 40.91
ceval-basic_medicine c409d6 accuracy gen 68.42
ceval-clinical_medicine 49e82d accuracy gen 31.82
ceval-urban_and_rural_planner 95b885 accuracy gen 47.83
ceval-accountant 002837 accuracy gen 36.73
ceval-fire_engineer bc23f5 accuracy gen 38.71
ceval-environmental_impact_assessment_engineer c64e2d accuracy gen 51.61
ceval-tax_accountant 3a5e3c accuracy gen 36.73
ceval-physician 6e277d accuracy gen 42.86
ceval-stem - naive_average gen 39.21
ceval-social-science - naive_average gen 57.43
ceval-humanities - naive_average gen 50.23
ceval-other - naive_average gen 44.62
ceval-hard - naive_average gen 32
ceval - naive_average gen 46.19
