【Agentic RL 智能体强化学习】第 48 篇综合实战一 — 训练数学推理模型DeepSeek R1 范式复现本系列定位从强化学习数学基础出发系统讲解 LLM 对齐RLHF/DPO/GRPO到 Agentic RL多轮智能体强化学习的全链路以 OpenRLHF 框架为实战主线。本篇你将学到端到端训练数学推理模型的完整流程数据准备DAPO-Math-17k 数据集处理RLVR 奖励函数LaTeX 答案精确匹配训练配置与监控GSM8K / MATH 基准测试一、项目概述本篇将把前 47 篇的知识综合到一个完整项目中——用 OpenRLHF 训练 Qwen3-4B 获得强数学推理能力。Qwen3-4B-Thinking基座模型准备数学数据DAPO-Math-17k编写 RLVR奖励函数配置训练参数REINFORCE-baseline启动训练OpenRLHF Ray vLLM监控与调优Wandb评估GSM8K / MATH 数学推理模型二、数据准备2.1 数据集DAPO-Math-17k属性值来源zhuzilin/dapo-math-17k样本数~17,000格式{prompt, label}特点高质量数学问题附标准答案2.2 数据格式# 原始数据格式{prompt:计算 $\\int_0^1 x^2 dx$ 的值。,label:\\frac{1}{3}}# OpenRLHF 处理后--data.prompt_dataset zhuzilin/dapo-math-17k--data.input_key prompt--data.label_key label--data.apply_chat_template三、RLVR 奖励函数# math_reward_func.pyimporttorchimportrefromsympyimportsimplify,Eq,sympifyfromsympy.parsing.lateximportparse_latexdefextract_answer(text):从模型输出中提取 \\boxed{} 答案# 匹配 \boxed{...}patterns[r\\boxed\{([^}])\},r答案是\s*[:]?\s*(.?)(?:\n|$),ranswer\s*[:]?\s*(.?)(?:\n|$),]forpatterninpatterns:matchre.search(pattern,text,re.DOTALL)ifmatch:returnmatch.group(1).strip()returnNonedefnormalize_answer(answer):标准化答案格式ifanswerisNone:returnNone# 去除空格和多余符号answeranswer.strip().rstrip(.)# 尝试 LaTeX 解析try:exprparse_latex(answer)returnstr(simplify(expr))except:returnanswerdefmath_verify(predicted,ground_truth):验证数学答案是否正确pred_normnormalize_answer(predicted)gt_normnormalize_answer(ground_truth)ifpred_normisNone:returnFalse# 精确匹配ifpred_normgt_norm:returnTrue# 符号等价检查try:pred_exprsympify(pred_norm)gt_exprsympify(gt_norm)ifsimplify(pred_expr-gt_expr)0:returnTrueexcept:pass# 数值比较try:ifabs(float(pred_norm)-float(gt_norm))1e-6:returnTrueexcept:passreturnFalsedefreward_func(queries,prompts,labels): RLVR 数学奖励函数 batch_sizelen(queries)rewardstorch.zeros(batch_size)correct_count0answered_count0fori,(query,label)inenumerate(zip(queries,labels)):# 提取模型答案predictedextract_answer(query)ifpredictedisnotNone:answered_count1ifmath_verify(predicted,label):rewards[i]1.0correct_count1accuracycorrect_count/batch_size answer_rateanswered_count/batch_sizereturn{rewards:rewards,scores:rewards,extra_logs:{accuracy:accuracy,answer_rate:answer_rate,}}四、训练配置# ray_cluster_setup.shray start--head--node-ip-address0.0.0.0 --num-gpus8# train_math_rlvr.shray job submit--addresshttp://127.0.0.1:8265\--runtime-env-json{working_dir: /openrlhf}\-- python3-mopenrlhf.cli.train_ppo_ray\--actor.model_name_or_pathQwen/Qwen3-4B-Thinking-2507\--reward.remote_urlexamples/python/math_reward_func.py\--data.prompt_datasetzhuzilin/dapo-math-17k\--data.input_keyprompt\--data.label_keylabel\--data.apply_chat_template\\--ref.num_nodes1--ref.num_gpus_per_node4\--actor.num_nodes1--actor.num_gpus_per_node4\--vllm.num_engines2--vllm.tensor_parallel_size2\--train.colocate_all\--vllm.gpu_memory_utilization0.7\--vllm.enable_sleep--ds.enable_sleep\--vllm.sync_backendnccl--vllm.enforce_eager\\--algo.advantage.estimatorreinforce_baseline\--algo.kl.use_loss--algo.kl.estimatork2\--algo.kl.init_coef1e-5\--actor.entropy_coef0.0\--algo.advantage.is_correction_enable\--algo.advantage.is_correction_typeicepop\\--rollout.batch_size128\--rollout.n_samples_per_prompt8\--train.batch_size1024\--algo.dynamic_filtering_enable\--algo.dynamic_filtering_range0.01.0\--train.dynamic_batch_enable\--data.max_len74240\--rollout.max_new_tokens64000\\--ds.zero_stage3--ds.param_dtypebf16\--actor.gradient_checkpointing_enable\--actor.adam.lr5e-7\--ckpt.output_dir./exp/Qwen3-4B-Math-RLVR\--logger.wandb.projectagentic-rl-math五、训练监控指标指标含义期望趋势reward/mean平均奖励↑reward/accuracy答题正确率↑reward/answer_rate有效回答率→ 或 ↑response_length/mean平均回答长度可能 ↑推理链变长kl/meanKL 散度稳定response_length/std回答长度方差可能 ↑多样性六、评估# GSM8K 评估python eval_math.py\--model./exp/Qwen3-4B-Math-RLVR\--benchmarkgsm8k\--n_samples1# MATH 评估python eval_math.py\--model./exp/Qwen3-4B-Math-RLVR\--benchmarkmath\--n_samples16.1 预期结果模型GSM8KMATHQwen3-4B-Base~70%~35%Qwen3-4B-Thinking~85%~50% RLVR 训练后~90%~60%本篇小结步骤关键点数据DAPO-Math-17k奖励LaTeX 精确匹配 符号验证算法REINFORCE±baseline ICEPOP部署Hybrid Engine 8 GPU监控accuracy↑, response_length↑评估GSM8K / MATH 基准下篇预告第 49 篇综合实战二 — 训练工具调用 Agent如果本篇内容对你有帮助欢迎点赞收藏有任何疑问欢迎在评论区交流。
