[Feature Request]: how to use megatron of ms-swift to finetune minicpm5-2b

[haodongze]

Feature request / 功能建议

how to use megatron of ms-swift to finetune minicpm5-2b

[hansjohn]

Hi, thanks for your interest in MiniCPM5-2B!

MiniCPM5-2B can be fine-tuned with Megatron-SWIFT. For better alignment with the MiniCPM5-2B training format, we recommend using the minicpm5_2b template available in ms-swift>=4.6.0.dev0.

The following installation example uses ms-swift==4.6.0.dev0:

git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
git checkout 654e24f17b5d9f40ed4b9ee4c56a723320244db5
pip install -e .
pip install "transformers>=5.6,<5.17"
pip install mcore-bridge==1.6.4

For MiniCPM5-2B, use:

megatron sft \
    --model /path/to/MiniCPM5-2B \
    --model_type llama \
    --template minicpm5_2b \
    --dataset /path/to/train.jsonl \
    --finetune true \
    --output_dir /path/to/output \
    ...

The important arguments are:

--model_type llama
--template minicpm5_2b

We have also updated the documentation with the ms-swift and Megatron-SWIFT configuration:

https://github.com/OpenBMB/MiniCPM/blob/main/docs/finetune/ms_swift.md

For the complete training options, please refer to the official ms-swift documentation:

• https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html
• https://swift.readthedocs.io/en/latest/Megatron-SWIFT/Quick-start.html