大模型之基于TRL的DPO对齐训练实战篇

1、概念说明1TRLTRL Transformer Reinforcement Learning它是huggingface提供的专门给Transformer大模型做对齐训练的工具库把大模型对齐常用算法封装好了。可以完成SFT 监督微调RM 奖励模型训练PPORLHFDPO / IPO / KTO 偏好对齐核心Trainer类SFTTrainer监督微调 SFT做 SFT 训练RewardTrainer训练奖励模型 RMPPOTrainer传统 RLHF 的 PPO 训练器需要 RM实现 PPO 强化学习循环会在线 generate 采样DPOTrainer实现标准 DPO 算法DPOTrainer的底层工作读取prompt/chosen/rejected按照 tokenizer 的 chat_template 拼接输入自动计算 policy 模型、ref 参考模型对chosen、rejected完整回答的序列联合对数概率实现完整 DPO loss自动处理 ref 模型冻结ref 不计算梯度支持 LoRA 训练peft不用全量微调自动计算监控指标rewards/chosen、rewards/rejected、奖励差值训练日志直接打印支持梯度累积、bf16、8bit 优化器等。总结TRL 是 HuggingFace 开源的大模型对齐库封装了 SFT、奖励模型、PPO、DPO 等对齐算法。我们做 DPO 直接使用DPOTrainer它内部已经实现 DPO 损失函数负责计算 policy/ref 模型的序列对数概率、冻结 ref 模型、训练循环极大降低对齐代码开发量。2、数据准备医疗循证DPO数据集还是医疗相关的DPO数据集因为之前已经训练了一个医疗相关的SFT模型。总共有1400条数据每条数据包含三个字段prompt(string): 医学问题或查询chosen(string): 高质量回答作为偏好目标rejected(string): 低质量回答作为负样本示例格式{ prompt: 在阿尔茨海默病与溃疡性结肠炎患者中PPARG 和 NOS2 作为共同基因是否通过调控巨噬细胞和小胶质细胞极化参与疾病的发生发展, chosen: 从目前的人类与动物实验证据来看PPARG 和 NOS2 很有可能作为共同炎症枢纽基因通过调控巨噬细胞/小胶质细胞的极化状态参与阿尔茨海默病和溃疡性结肠炎的发生发展..., rejected: 这是一个非常具体且专业的问题涉及到两种疾病的共同机制。根据现有的生物医学研究我们可以进行一个基于科学逻辑的推理和分析... }这个数据集的标注逻辑chosen循证严谨区分证据等级承认哪些地方证据不足rejected过于绝对把未完全证实的假说当成板上钉钉结论产生误导。共有1400条样本集人工切分成2部分med_dpo_answer_train.jsonl和med_dpo_answer_test.jsonl3、训练dpo的lora代码import os os.environ[PYTORCH_CUDA_ALLOC_CONF] expandable_segments:True import torch from datasets import load_dataset from transformers import AutoModelForCausalLM, AutoTokenizer from trl import DPOTrainer, DPOConfig from peft import LoraConfig # 路径配置 model_path ./merged_sft_qwen7b_med output_lora_dir ./med_dpo_lora train_file /root/autodl-tmp/datas/rlhf/med_dpo_answer_train.jsonl test_file /root/autodl-tmp/datas/rlhf/med_dpo_answer_test.jsonl SYSTEM_PROMPT 你是专业的医疗咨询助手回答仅供科普参考不能替代执业医师面诊诊疗请遵从线下医生的专业意见。 # 加载tokenizer tokenizer AutoTokenizer.from_pretrained(model_path, trust_remote_codeTrue) # 加载数据集 ds load_dataset( json, data_files{ train: train_file, test: test_file } ) def add_system_template(sample): messages [ {role: system, content: SYSTEM_PROMPT}, {role: user, content: sample[prompt]} ] full_prompt tokenizer.apply_chat_template( messages, tokenizeFalse, add_generation_promptTrue ) return { prompt: full_prompt, chosen: sample[chosen], rejected: sample[rejected] } ds ds.map(add_system_template) # 预先过滤超长样本总token不超过8192 def filter_fn(sample): full_text sample[prompt] sample[chosen] sample[rejected] tok tokenizer(full_text, return_lengthTrue) return tok[length][0] 8192 ds ds.filter(filter_fn) train_dataset ds[train].shuffle(seed42) eval_dataset ds[test] #打印校验 print(检查第一条样本prompt头部) print(ds[train][0][prompt][:600]) print(f训练集样本数{len(train_dataset)}) print(f验证集样本数{len(eval_dataset)}) # 加载完整SFT基座模型 model AutoModelForCausalLM.from_pretrained( model_path, torch_dtypetorch.bfloat16, device_mapauto, trust_remote_codeTrue ) # 构造LoraConfig对象 lora_config LoraConfig( r16, lora_alpha32, target_modules[q_proj, k_proj, v_proj, o_proj], lora_dropout0.05, biasnone, task_typeCAUSAL_LM ) # trl1.9.2 DPOConfig 开启 do_evalTrue dpo_args DPOConfig( output_diroutput_lora_dir, beta0.25, loss_typesigmoid, learning_rate4e-6, num_train_epochs0.7, # 1100样本控制不要过高防止过拟合 per_device_train_batch_size2, gradient_accumulation_steps4, logging_steps5, # --------eval相关配置-------- do_evalTrue, eval_strategysteps, eval_steps10, # 每10训练step跑一次验证集 load_best_model_at_endTrue, metric_for_best_modeleval_rewards/accuracies, greater_is_betterTrue, # 奖励准确率越高越好 save_total_limit3, # 最多保留3个checkpoint bf16True, optimadamw_torch, report_to[], ) trainer DPOTrainer( modelmodel, argsdpo_args, train_datasettrain_dataset, eval_dataseteval_dataset, peft_configlora_config, ref_modelNone ) #启动训练 trainer.train() # load_best_model_at_end开启后trainer.model已经是验证集最优权重 trainer.save_model(output_lora_dir) print(fDPO训练完成最优LoRA适配器输出至{output_lora_dir})运行环境使用32*4128G的显存运行的32*2运行不起来在eval阶段再加载eval模型时会出现OOM。4、运行指标rootautodl-container-45404b8e68-e4244a4c:~/autodl-tmp/codes/sft# python train_dpo_med.py 检查第一条样本prompt头部 |im_start|system 你是专业的医疗咨询助手回答仅供科普参考不能替代执业医师面诊诊疗请遵从线下医生的专业意见。|im_end| |im_start|user 基于最新循证指南的综合治疗管理对于肝硬化患者的临床预后和并发症控制有何影响|im_end| |im_start|assistant 训练集样本数1100 验证集样本数299 [transformers] torch_dtype is deprecated! Use dtype instead! Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 339/339 [00:0200:00, 143.71it/s] Dropping fully truncated examples from train dataset: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 1100/1100 [00:0100:00, 781.46 examples/s] Dropping fully truncated examples from eval dataset: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 299/299 [00:0000:00, 754.57 examples/s] [transformers] The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizers values. Updated tokens: {bos_token_id: None, pad_token_id: 151643}. {loss: 0.6655, grad_norm: 22.51, learning_rate: 3.835e-06, entropy: 1.491, num_tokens: 8.12e04, logits/chosen: 0.8174, logits/rejected: 1.127, mean_token_accuracy: 0.5815, rewards/chosen: -0.118, rewards/rejected: -0.218, rewards/accuracies: 0.575, rewards/margins: 0.09999, logps/chosen: -1649, logps/rejected: -1457, epoch: 0.03636} {loss: 0.692, grad_norm: 23.94, learning_rate: 3.629e-06, entropy: 1.462, num_tokens: 1.613e05, logits/chosen: 0.8175, logits/rejected: 1.096, mean_token_accuracy: 0.5936, rewards/chosen: -0.147, rewards/rejected: -0.2055, rewards/accuracies: 0.55, rewards/margins: 0.05849, logps/chosen: -1611, logps/rejected: -1394, epoch: 0.07273} {eval_loss: 0.6538, eval_runtime: 263.1, eval_samples_per_second: 1.136, eval_steps_per_second: 0.144, eval_entropy: 1.456, eval_num_tokens: 1.613e05, eval_logits/chosen: 0.8848, eval_logits/rejected: 1.132, eval_mean_token_accuracy: 0.5933, eval_rewards/chosen: -0.2268, eval_rewards/rejected: -0.3615, eval_rewards/accuracies: 0.6118, eval_rewards/margins: 0.1347, eval_logps/chosen: -1583, eval_logps/rejected: -1398, epoch: 0.07273} {loss: 0.6776, grad_norm: 26.73, learning_rate: 3.423e-06, entropy: 1.461, num_tokens: 2.426e05, logits/chosen: 0.8494, logits/rejected: 1.165, mean_token_accuracy: 0.5897, rewards/chosen: -0.2107, rewards/rejected: -0.3011, rewards/accuracies: 0.6, rewards/margins: 0.09039, logps/chosen: -1606, logps/rejected: -1445, epoch: 0.1091} {loss: 0.5729, grad_norm: 25.62, learning_rate: 3.216e-06, entropy: 1.433, num_tokens: 3.223e05, logits/chosen: 0.8954, logits/rejected: 1.113, mean_token_accuracy: 0.5907, rewards/chosen: -0.2589, rewards/rejected: -0.5807, rewards/accuracies: 0.725, rewards/margins: 0.3218, logps/chosen: -1577, logps/rejected: -1392, epoch: 0.1455} {eval_loss: 0.5215, eval_runtime: 262.6, eval_samples_per_second: 1.139, eval_steps_per_second: 0.145, eval_entropy: 1.455, eval_num_tokens: 3.223e05, eval_logits/chosen: 0.8848, eval_logits/rejected: 1.131, eval_mean_token_accuracy: 0.5935, eval_rewards/chosen: -0.3105, eval_rewards/rejected: -0.7597, eval_rewards/accuracies: 0.8158, eval_rewards/margins: 0.4492, eval_logps/chosen: -1584, eval_logps/rejected: -1400, epoch: 0.1455} {loss: 0.4748, grad_norm: 19.57, learning_rate: 3.01e-06, entropy: 1.448, num_tokens: 4.024e05, logits/chosen: 0.8937, logits/rejected: 1.151, mean_token_accuracy: 0.5905, rewards/chosen: -0.4272, rewards/rejected: -0.9954, rewards/accuracies: 0.875, rewards/margins: 0.5682, logps/chosen: -1604, logps/rejected: -1396, epoch: 0.1818} {loss: 0.4201, grad_norm: 15.62, learning_rate: 2.804e-06, entropy: 1.436, num_tokens: 4.829e05, logits/chosen: 0.931, logits/rejected: 1.146, mean_token_accuracy: 0.5934, rewards/chosen: -0.5315, rewards/rejected: -1.295, rewards/accuracies: 0.875, rewards/margins: 0.7631, logps/chosen: -1586, logps/rejected: -1374, epoch: 0.2182} {eval_loss: 0.3951, eval_runtime: 261.7, eval_samples_per_second: 1.143, eval_steps_per_second: 0.145, eval_entropy: 1.454, eval_num_tokens: 4.829e05, eval_logits/chosen: 0.8867, eval_logits/rejected: 1.134, eval_mean_token_accuracy: 0.5934, eval_rewards/chosen: -0.5125, eval_rewards/rejected: -1.343, eval_rewards/accuracies: 0.9046, eval_rewards/margins: 0.8301, eval_logps/chosen: -1584, eval_logps/rejected: -1402, epoch: 0.2182} {loss: 0.3736, grad_norm: 16.5, learning_rate: 2.598e-06, entropy: 1.458, num_tokens: 5.635e05, logits/chosen: 0.8576, logits/rejected: 1.14, mean_token_accuracy: 0.5949, rewards/chosen: -0.556, rewards/rejected: -1.466, rewards/accuracies: 0.95, rewards/margins: 0.91, logps/chosen: -1603, logps/rejected: -1409, epoch: 0.2545} {loss: 0.3498, grad_norm: 16.54, learning_rate: 2.392e-06, entropy: 1.458, num_tokens: 6.434e05, logits/chosen: 0.8618, logits/rejected: 1.138, mean_token_accuracy: 0.5927, rewards/chosen: -0.726, rewards/rejected: -1.732, rewards/accuracies: 0.925, rewards/margins: 1.005, logps/chosen: -1599, logps/rejected: -1395, epoch: 0.2909} {eval_loss: 0.3013, eval_runtime: 262.2, eval_samples_per_second: 1.14, eval_steps_per_second: 0.145, eval_entropy: 1.454, eval_num_tokens: 6.434e05, eval_logits/chosen: 0.888, eval_logits/rejected: 1.136, eval_mean_token_accuracy: 0.5935, eval_rewards/chosen: -0.7372, eval_rewards/rejected: -1.949, eval_rewards/accuracies: 0.9605, eval_rewards/margins: 1.212, eval_logps/chosen: -1585, eval_logps/rejected: -1405, epoch: 0.2909} {loss: 0.2906, grad_norm: 13.11, learning_rate: 2.186e-06, entropy: 1.433, num_tokens: 7.24e05, logits/chosen: 0.8954, logits/rejected: 1.148, mean_token_accuracy: 0.5962, rewards/chosen: -0.851, rewards/rejected: -2.113, rewards/accuracies: 0.95, rewards/margins: 1.262, logps/chosen: -1572, logps/rejected: -1436, epoch: 0.3273} {loss: 0.2298, grad_norm: 13.48, learning_rate: 1.979e-06, entropy: 1.424, num_tokens: 8.042e05, logits/chosen: 0.8616, logits/rejected: 1.15, mean_token_accuracy: 0.5913, rewards/chosen: -0.791, rewards/rejected: -2.354, rewards/accuracies: 0.975, rewards/margins: 1.563, logps/chosen: -1607, logps/rejected: -1360, epoch: 0.3636} {eval_loss: 0.2059, eval_runtime: 262.4, eval_samples_per_second: 1.14, eval_steps_per_second: 0.145, eval_entropy: 1.454, eval_num_tokens: 8.042e05, eval_logits/chosen: 0.8875, eval_logits/rejected: 1.139, eval_mean_token_accuracy: 0.5934, eval_rewards/chosen: -0.8336, eval_rewards/rejected: -2.586, eval_rewards/accuracies: 0.977, eval_rewards/margins: 1.752, eval_logps/chosen: -1586, eval_logps/rejected: -1407, epoch: 0.3636} {loss: 0.222, grad_norm: 8.034, learning_rate: 1.773e-06, entropy: 1.475, num_tokens: 8.835e05, logits/chosen: 0.8554, logits/rejected: 1.093, mean_token_accuracy: 0.5906, rewards/chosen: -1.095, rewards/rejected: -2.744, rewards/accuracies: 0.975, rewards/margins: 1.65, logps/chosen: -1617, logps/rejected: -1368, epoch: 0.4} {loss: 0.1913, grad_norm: 9.952, learning_rate: 1.567e-06, entropy: 1.445, num_tokens: 9.632e05, logits/chosen: 0.7705, logits/rejected: 1.081, mean_token_accuracy: 0.594, rewards/chosen: -1.036, rewards/rejected: -3.077, rewards/accuracies: 0.975, rewards/margins: 2.041, logps/chosen: -1596, logps/rejected: -1351, epoch: 0.4364} {eval_loss: 0.1529, eval_runtime: 262.2, eval_samples_per_second: 1.14, eval_steps_per_second: 0.145, eval_entropy: 1.453, eval_num_tokens: 9.632e05, eval_logits/chosen: 0.8913, eval_logits/rejected: 1.142, eval_mean_token_accuracy: 0.5931, eval_rewards/chosen: -1.181, eval_rewards/rejected: -3.39, eval_rewards/accuracies: 0.9803, eval_rewards/margins: 2.209, eval_logps/chosen: -1587, eval_logps/rejected: -1410, epoch: 0.4364} {loss: 0.1591, grad_norm: 4.528, learning_rate: 1.361e-06, entropy: 1.422, num_tokens: 1.042e06, logits/chosen: 0.8682, logits/rejected: 1.072, mean_token_accuracy: 0.6012, rewards/chosen: -1.066, rewards/rejected: -3.329, rewards/accuracies: 0.975, rewards/margins: 2.263, logps/chosen: -1560, logps/rejected: -1357, epoch: 0.4727} {loss: 0.1131, grad_norm: 7.361, learning_rate: 1.155e-06, entropy: 1.413, num_tokens: 1.122e06, logits/chosen: 0.8699, logits/rejected: 1.129, mean_token_accuracy: 0.6021, rewards/chosen: -1.194, rewards/rejected: -3.633, rewards/accuracies: 1, rewards/margins: 2.439, logps/chosen: -1548, logps/rejected: -1421, epoch: 0.5091} {eval_loss: 0.1244, eval_runtime: 262.3, eval_samples_per_second: 1.14, eval_steps_per_second: 0.145, eval_entropy: 1.454, eval_num_tokens: 1.122e06, eval_logits/chosen: 0.8879, eval_logits/rejected: 1.143, eval_mean_token_accuracy: 0.5932, eval_rewards/chosen: -0.9224, eval_rewards/rejected: -3.426, eval_rewards/accuracies: 0.9836, eval_rewards/margins: 2.503, eval_logps/chosen: -1586, eval_logps/rejected: -1411, epoch: 0.5091} {loss: 0.1075, grad_norm: 6.041, learning_rate: 9.485e-07, entropy: 1.464, num_tokens: 1.201e06, logits/chosen: 0.8079, logits/rejected: 1.103, mean_token_accuracy: 0.5914, rewards/chosen: -0.9542, rewards/rejected: -3.529, rewards/accuracies: 0.975, rewards/margins: 2.574, logps/chosen: -1566, logps/rejected: -1403, epoch: 0.5455} {loss: 0.1837, grad_norm: 6.479, learning_rate: 7.423e-07, entropy: 1.419, num_tokens: 1.279e06, logits/chosen: 0.8548, logits/rejected: 1.123, mean_token_accuracy: 0.6027, rewards/chosen: -1.186, rewards/rejected: -3.573, rewards/accuracies: 0.95, rewards/margins: 2.387, logps/chosen: -1561, logps/rejected: -1292, epoch: 0.5818} {eval_loss: 0.1107, eval_runtime: 262.5, eval_samples_per_second: 1.139, eval_steps_per_second: 0.145, eval_entropy: 1.455, eval_num_tokens: 1.279e06, eval_logits/chosen: 0.8841, eval_logits/rejected: 1.141, eval_mean_token_accuracy: 0.5932, eval_rewards/chosen: -0.9596, eval_rewards/rejected: -3.634, eval_rewards/accuracies: 0.9836, eval_rewards/margins: 2.675, eval_logps/chosen: -1586, eval_logps/rejected: -1411, epoch: 0.5818} {loss: 0.07903, grad_norm: 6.533, learning_rate: 5.361e-07, entropy: 1.464, num_tokens: 1.359e06, logits/chosen: 0.8634, logits/rejected: 1.161, mean_token_accuracy: 0.5882, rewards/chosen: -0.9219, rewards/rejected: -3.809, rewards/accuracies: 1, rewards/margins: 2.887, logps/chosen: -1603, logps/rejected: -1392, epoch: 0.6182} {loss: 0.07335, grad_norm: 5.129, learning_rate: 3.299e-07, entropy: 1.433, num_tokens: 1.44e06, logits/chosen: 0.8821, logits/rejected: 1.128, mean_token_accuracy: 0.6019, rewards/chosen: -1.154, rewards/rejected: -4.002, rewards/accuracies: 1, rewards/margins: 2.847, logps/chosen: -1560, logps/rejected: -1455, epoch: 0.6545} {eval_loss: 0.09361, eval_runtime: 262.4, eval_samples_per_second: 1.14, eval_steps_per_second: 0.145, eval_entropy: 1.455, eval_num_tokens: 1.44e06, eval_logits/chosen: 0.8885, eval_logits/rejected: 1.142, eval_mean_token_accuracy: 0.5931, eval_rewards/chosen: -1.093, eval_rewards/rejected: -4.037, eval_rewards/accuracies: 0.9868, eval_rewards/margins: 2.943, eval_logps/chosen: -1587, eval_logps/rejected: -1413, epoch: 0.6545} {loss: 0.07379, grad_norm: 4.132, learning_rate: 1.237e-07, entropy: 1.446, num_tokens: 1.52e06, logits/chosen: 0.8279, logits/rejected: 1.091, mean_token_accuracy: 0.5916, rewards/chosen: -1.101, rewards/rejected: -4.017, rewards/accuracies: 1, rewards/margins: 2.916, logps/chosen: -1606, logps/rejected: -1405, epoch: 0.6909} {eval_loss: 0.09319, eval_runtime: 262.5, eval_samples_per_second: 1.139, eval_steps_per_second: 0.145, eval_entropy: 1.454, eval_num_tokens: 1.552e06, eval_logits/chosen: 0.8901, eval_logits/rejected: 1.146, eval_mean_token_accuracy: 0.5931, eval_rewards/chosen: -0.9987, eval_rewards/rejected: -4.021, eval_rewards/accuracies: 0.9868, eval_rewards/margins: 3.022, eval_logps/chosen: -1586, eval_logps/rejected: -1413, epoch: 0.7055} {train_runtime: 3878, train_samples_per_second: 0.199, train_steps_per_second: 0.025, train_loss: 0.3075, epoch: 0.7055} 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 97/97 [1:04:3800:00, 39.98s/it] DPO训练完成最优LoRA适配器输出至./med_dpo_lora1rewards/accuracies含义偏好判断准确率2rewards/margins含义奖励差值margin注意DPO reward 带 \(\boldsymbol{\beta}\) 缩放所以 reward 是相对值不是 0‑1 分数。margin 0模型认为 chosen 更好margin 越大模型认为好坏差距越大。rewards/margins训练集平均 margineval_rewards/margins验证集平均 margin3lossloss训练集上 DPO 损失的批次均值eval_loss验证集上 DPO 损失均值4entropy含义对每一步解码位置模型输出词表维度概率分布 p计算信息熵训练时对一批样本做平均。5、测试推理代码import torch from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel # 路径配置与训练脚本完全对齐无需修改 BASE_MODEL_PATH ./merged_sft_qwen7b_med DPO_LORA_PATH ./med_dpo_lora # 医学固定系统提示词和训练一致 SYSTEM_PROMPT 你是专业的医疗咨询助手回答仅供科普参考不能替代执业医师面诊诊疗请遵从线下医生的专业意见。 # 测试问题全部为【训练集外】通用医学问题检测泛化能力 TEST_QUESTIONS [ 高血压患者日常饮食需要注意哪些事项, 糖尿病患者如何科学控制餐后血糖, 肝硬化患者常见的并发症有哪些日常如何预防, 慢性支气管炎患者秋冬季节如何养护, 高血脂长期不控制会对身体造成哪些危害 ] # 加载模型 print(正在加载SFT基座模型...) tokenizer AutoTokenizer.from_pretrained(BASE_MODEL_PATH, trust_remote_codeTrue) base_model AutoModelForCausalLM.from_pretrained( BASE_MODEL_PATH, torch_dtypetorch.bfloat16, device_mapauto, trust_remote_codeTrue ) print(正在加载DPO-LoRA微调模型...) dpo_model PeftModel.from_pretrained(base_model, DPO_LORA_PATH) # 推理生成函数 def generate_answer(model, query): messages [ {role: system, content: SYSTEM_PROMPT}, {role: user, content: query} ] input_text tokenizer.apply_chat_template( messages, tokenizeFalse, add_generation_promptTrue ) inputs tokenizer(input_text, return_tensorspt).to(cuda) # 通用稳定生成参数 outputs model.generate( **inputs, max_new_tokens1024, temperature0.7, top_p0.9, do_sampleTrue, eos_token_idtokenizer.eos_token_id ) return tokenizer.decode(outputs[0][inputs[input_ids].shape[-1]:], skip_special_tokensTrue) # 批量对比测试 if __name__ __main__: for idx, question in enumerate(TEST_QUESTIONS, 1): print(f\n{ * 80}) print(f【测试问题 {idx}】{question}) print(f{ * 80}) print(\n[1] 【原始SFT基座模型回答】) print(generate_answer(base_model, question)) print(\n[2] 【SFT DPO-LoRA 微调模型回答】) print(generate_answer(dpo_model, question))