Preparing an Environment for Fine-tuning Qwen 2.5

This article outlines the steps to prepare a Conda environment for fine-tuning the large language model Qwen 2.5 with LLaMA-Factory on a Ubuntu 20.04.6 LTS cluster with eight NVIDIA GeForce RTX 3090 processors. The commit used is af336275021cd6aee3fe9f67b9ac9bcd1276de7c.

The very first step is to get the correct version of PyTorch (2.3.1) installed from the PyTorch channel. All three packages (PyTorch, torchaudio and torchvision) must come from the same channel to ensure that they are compatible with each other. If PyTorch is from conda-forge and torchaudio, which is missing from the conda-forge channel, is from PyTorch, importing torchaudio will fail, complaining about a missing symbol in libtorch. PyTorch had better be installed with the creation of the environment.

conda create -c pytorch -c nvidia -n sft python=3.11 pytorch=2.3.1 torchvision=0.18.1 torchaudio=2.3.1 pytorch-cuda=12.1
conda activate sft

The name I use here is sft.

Next, we install the other packages. Do not bother with the optional ones; they help little and may cause you much headache. Note that LLaMA-Factory is perhaps unnecessarily stringent about the versions of the dependencies.

conda install -c conda-forge transformers=4.46.1 datasets=3.1.0 accelerate=1.0.1 peft=0.12.0 trl=0.9.6 'tyro<0.9.0' 'gradio<5.0.0' nltk sacremoses sacrebleu scipy einops sentencepiece tiktoken protobuf uvicorn pydantic fastapi sse-starlette matplotlib fire packaging pyyaml av

The final step is to install LLaMA-Factory itself from source.

git clone [email protected]:hiyouga/LLaMA-Factory.git --depth 1
cd LLaMA-Factory
git checkout af336275021cd6aee3fe9f67b9ac9bcd1276de7c
pip install --no-deps -e .

Verify the setup with:

llamafactory-cli version

The expected output is:

----------------------------------------------------------
| Welcome to LLaMA Factory, version 0.9.2.dev0           |
|                                                        |
| Project page: https://github.com/hiyouga/LLaMA-Factory |
----------------------------------------------------------

Now try fine-tuning Qwen/Qwen2.5-0.5B-Instruct with LoRA. For me training on 8,000 examples for three epochs takes about 16 minutes. Here's a link to the official tutorial.

If everything fails, try installing from this exported environment:

conda env create -f SFT-Environment.yml
conda activate sft
# Go to the LLaMA-Factory directory
pip install --no-deps -e .