Ref欢迎来到 EvalScope 中文教程 | EvalScope支持的数据集 | EvalScopehttps://github.com/EleutherAI/lm-evaluation-harness如何使用lm-evaluation-harness零代码评估大模型【GPT】中文大语言模型梳理与测评C-Eval 、AGIEval、MMLU、SuperCLUE_superclue c-eval哪个更权威-CSDN博客Note: 本文主要针对评测本地部署的vllm/sglang部署。evalscope使用lmeval评测sglang本地部署依赖vllm导致有一些问题推荐使用evalscope。pip install evalscope --upgrade -i https://pypi.tuna.tsinghua.edu.cn/simple pip install evalscope[perf] -i https://pypi.tuna.tsinghua.edu.cn/simple评测sglang本地部署evalscope eval \ --model DeepSeek-V3.1-Terminus \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets mmlu \ --dataset-args {mmlu: {subset_list: [high_school_physics, high_school_psychology], few_shot_num: 5}} \ --eval-batch-size 64只需要评测子任务的话加上--dataset-args。evalscope eval \ --model DeepSeek-V3.2 \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets aime25 \ --dataset-hub huggingface \ --eval-batch-size 128 evalscope eval \ --model DeepSeek-V3.2 \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets bbh \ --dataset-args {bbh: {subset_list: [boolean_expressions], few_shot_num: 3}} \ --eval-batch-size 64设置数据集本地路径--dataset-args {bbh: {local_path: /data1/dataset/bbh}} \generation-config--generation-config {do_sample:true,temperature:0.5,chat_template_kwargs:{enable_thinking:false}}# DeepSeek-V3.2 --generation-config {chat_template_kwargs:{thinking: true}}# subset_list: chinese, english evalscope eval \ --model DeepSeek-V3.2 \ --api-url http://xxx_ip:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets needle_haystack \ --dataset-args {needle_haystack: {subset_list: [chinese], local_path: /data1/dataset/Needle-in-a-Haystack-Corpus}} \ --judge-strategy auto \ --judge-worker-num 80 \ --judge-model-args {api_url: http://judge_model_ip:30000/v1, model_id: DeepSeek-V3.2} \ --eval-batch-size 80LongBench-v2LongBench-v2 | EvalScopeLongBench-v2各子集统计数据子集样本数提示词平均长度提示词最小长度提示词最大长度short180124200.4249433841252medium215501002.721721082233351long1082861217.9472082316184015evalscope eval \ --model GLM-5.3-Flash \ --api-url http://localhost:12121/v1 \ --api-key EMPTY \ --eval-type openai_api \ --generation-config timeout1800 \ --datasets longbench_v2 \ --dataset-args {longbench_v2: {subset_list: [short], local_path: /data1/datasets/llm_dataset/llm_accuracy_bench/LongBench-v2/}} \ --generation-config {max_tokens:24576} \ --eval-batch-size 20 \ --limit 100sglang自带评测gsm8kpython benchmark/gsm8k/bench_sglang.py --port 30000 --num-shots 5 --num-questions 500 python benchmark/gsm8k/bench_sglang.py --host http://127.0.0.1 --port 30000 --num-shots 5 --num-questions 500使用lm_evallm_eval安装# pip install lm-eval pip install lm-eval[api]源码安装git clone https://github.com/EleutherAI/lm-evaluation-harness cd lm-evaluation-harness pip install -e .查看支持的评测任务lm-eval --tasks listVLLM/SGLang serving API评测lm_eval \ --model local-completions \ --tasks mmlu \ --batch_size8 \ --model_args {model: Qwen/Qwen3-8B-FP8, base_url: http://localhost:8000/v1/completions, num_concurrent: 8}这个可以评测VLLM/SGLang启动的serving服务提供的api接口从而评测VLLM/sglang不同部署方案的效果。MMLU除了mmlu整体评测还可以分为4个子任务单独评测mmlu_stemmmlu_othermmlu_social_sciencesmmlu_humanities或者更精细的子任务评测。
