I did SFT on a 32B model. The config 1 outperforms config 2 by a margin on BLUE score.…
- User
I did SFT on a 32B model. The config 1 outperforms config 2 by a margin on BLUE score. Explain why and also how can we improve further. Currently accuracy is only 7/10 for config 1 and 4/10 for config 2. Config 1:
bf16: true cutoff_len: 1024 dataset: acts_cot dataset_dir: data ddp_timeout: 180000000 do_train: true finetuning_type: lora flash_attn: auto gradient_accumulation_steps: 8 include_num_input_tokens_seen: true learning_rate: 5.0e-05 logging_steps: 5 lora_alpha: 16 lora_dropout: 0 lora_rank: 8 lora_target: all lr_scheduler_type: cosine max_grad_norm: 1.0 max_samples: 100000 model_name_or_path: Qwen/Qwen2.5-32B-Instruct num_train_epochs: 3.0 optim: adamw_torch output_dir: saves/Qwen2.5-32B-Instruct/lora/train_2024-09-24-17-38-00-32b-4bit packing: false per_device_train_batch_size: 8 plot_loss: true preprocessing_num_workers: 16 quantization_bit: 4 quantization_method: bitsandbytes report_to: none save_steps: 10 stage: sft template: qwen warmup_steps: 0Config 2:bf16: true cutoff_len: 1024 dataset: acts_cot dataset_dir: data ddp_timeout: 180000000 do_train: true enable_liger_kernel: true eval_steps: 10 eval_strategy: steps finetuning_type: lora flash_attn: auto gradient_accumulation_steps: 8 include_num_input_tokens_seen: true learning_rate: 5.0e-06 logging_steps: 5 lora_alpha: 16 lora_dropout: 0 lora_rank: 8 lora_target: q_proj,v_proj lr_scheduler_type: cosine max_grad_norm: 1.0 max_samples: 500000 model_name_or_path: Qwen/Qwen2.5-32B-Instruct num_train_epochs: 3.0 optim: adamw_torch output_dir: saves/Qwen2.5-32B-Instruct/lora/train_2024-09-25-19-48-21 packing: false per_device_eval_batch_size: 24 per_device_train_batch_size: 24 plot_loss: true preprocessing_num_workers: 16 quantization_bit: 4 quantization_method: bitsandbytes report_to: none save_steps: 10 stage: sft template: qwen val_size: 0.05 warmup_steps: 10Train result for config 1:{ "epoch": 2.988480526604498, "num_input_tokens_seen": 22863968, "total_flos": 4.397072693194457e+18, "train_loss": 0.6360728469364093, "train_runtime": 30863.3065, "train_samples_per_second": 1.417, "train_steps_per_second": 0.022 }Train result for config 2:{ "epoch": 2.98961937716263, "num_input_tokens_seen": 24059336, "total_flos": 4.618482676450836e+18, "train_loss": 1.0954694322965763, "train_runtime": 27054.9013, "train_samples_per_second": 1.536, "train_steps_per_second": 0.008 } - Assistant
Based on the configurations and training results you've provided, Config 1 outperforms Config 2 on the BLUE score primarily due to a combination of factors related to learning rate, LoRA (Low-Rank Adaptation) settings, and training dynamics. Here's a detailed analysis of why Config 1 performed better and how you can further improve your fine-tuning process.
1. Key Differences Between Config 1 and Config 2
-
Learning Rate:
- Config 1:
learning_rate: 5.0e-05 - Config 2:
learning_rate: 5.0e-06
Impact:
- Config 1 uses a higher learning rate, which generally allows the model to learn more quickly and effectively during training.
- Config 2 has a significantly lower learning rate, which may result in slower learning and insufficient convergence within the same number of epochs.
- Config 1:
-
LoRA Targeting:
- Config 1:
lora_target: all - Config 2:
lora_target: q_proj,v_proj
Impact:
- Config 1 applies LoRA to all applicable layers, allowing for more comprehensive model adaptation.
- Config 2 restricts LoRA to only
q_projandv_projlayers, which may limit the model's adaptability and capacity to fine-tune effectively.
- Config 1:
-
Batch Size:
- Config 1:
per_device_train_batch_size: 8 - Config 2:
per_device_train_batch_size: 24
Impact:
- Config 1 has a smaller batch size, which can lead to more frequent weight updates and potentially better generalization.
- Config 2 uses a larger batch size, which might require adjustments in learning rate and could lead to poorer generalization if not tuned properly.
- Config 1:
-
Max Samples:
- Config 1:
max_samples: 100000 - Config 2:
max_samples: 500000
Impact:
- Config 1 trains on fewer samples but with a higher learning rate, which might be more effective if the additional samples in Config 2 are not contributing meaningful diversity or complexity to the training data.
- Config 1:
-
Warmup Steps:
- Config 1:
warmup_steps: 0 - Config 2:
warmup_steps: 10
Impact:
- Config 2 introduces a small number of warmup steps, which is generally good for stabilizing training, but in this case, it may not compensate for the lower learning rate.
- Config 1:
-
LoRA Parameters:
- Both configs use
lora_alpha: 16andlora_rank: 8, ensuring similar scaling for the LoRA layers.
- Both configs use
-
Additional Settings:
- Config 2 includes
enable_liger_kernel: true, which might introduce complexities or performance variations not present in Config 1.
- Config 2 includes
2. Analysis of Training Results
-
Config 1:
- Train Loss: 0.636
- Training Runtime: ~8.6 hours
- Train Samples/sec: 1.417
- Accuracy: 7/10
-
Config 2:
- Train Loss: 1.095
- Training Runtime: ~7.5 hours
- Train Samples/sec: 1.536
- Accuracy: 4/10
Interpretation:
- Config 1 achieved a significantly lower training loss, indicating better convergence.
- Despite processing fewer samples, Config 1's higher learning rate and broader LoRA application led to more effective learning.
- Config 2 struggled with higher loss and lower accuracy, likely due to its lower learning rate and more restricted LoRA targets.
3. Recommendations for Further Improvement
-
Optimize Learning Rate:
- Current Insight: Config 1's higher learning rate was beneficial.
- Action: Experiment with learning rates between
5.0e-05and5.0e-04to find an optimal balance that ensures effective learning without causing instability.
-
Adjust LoRA Targeting:
- Current Insight: Broad LoRA targeting in Config 1 was more effective.
- Action:
- Either continue with
lora_target: allor identify other critical layers beyondq_projandv_projto include. - Consider a staggered approach where you gradually increase the LoRA targets as training progresses.
- Either continue with
-
Batch Size Tuning:
- Current Insight: Smaller batch sizes in Config 1 may aid generalization.
- Action: Continue experimenting with batch sizes. You might find that a size between 8 and 24 offers a better trade-off between training speed and model performance.
-
Increase Training Epochs:
- Current Insight: Both configs are trained for ~3 epochs.
- Action: Extend the number of training epochs to allow the model more opportunities to learn, especially if you adjust the learning rate and batch size.
-
Implement Learning Rate Warmup:
- Current Insight: Config 1 has no warmup steps, while Config 2 has minimal warmup.
- Action: Introduce a more substantial warmup phase (e.g., 500-1000 steps) to stabilize training, especially if using higher learning rates.
-
Regular Evaluation:
- Current Insight: Only Config 2 has evaluation steps configured.
- Action: Implement regular evaluation checkpoints in Config 1 to monitor performance and prevent overfitting.
-
Data Quality and Diversity:
- Current Insight: Config 2 uses more samples, but it's unclear if they add value.
- Action: Ensure that the additional samples in larger datasets are of high quality and contribute meaningful diversity to the training process.
-
Utilize Advanced Optimization Techniques:
- Action: Experiment with techniques like gradient clipping, different optimizer settings (e.g., learning rate schedulers beyond cosine), or even mixed precision training if not already fully utilized.
-
Monitor Resource Allocation:
- Current Insight: Config 2 uses
enable_liger_kernel: true, which might affect performance. - Action: Assess the impact of such settings on training stability and performance. If they introduce complexities, consider disabling them unless they provide clear benefits.
- Current Insight: Config 2 uses
-
Hyperparameter Optimization:
- Action: Utilize automated hyperparameter tuning tools to systematically explore the hyperparameter space for optimal settings.
4. Summary
Config 1 outperformed Config 2 primarily because it utilized a higher learning rate and applied LoRA more broadly, leading to better model adaptation and convergence within the training epochs. To further improve your fine-tuning process:
- Fine-tune the learning rate to ensure optimal convergence.
- Broaden or strategically target LoRA layers to enhance model adaptability.
- Adjust batch sizes and training epochs to balance learning speed and generalization.
- Introduce more robust evaluation metrics and checkpoints to monitor training progress effectively.
By iteratively experimenting with these adjustments, you can enhance the performance of your 32B model beyond the current benchmarks.
-