Fine‑Tune a 7B LLM with LoRA on Hugging Face in One GPU

#Intro

In this tutorial we will fine‑tune a 7‑billion‑parameter LLaMA‑style model on a custom text dataset using Low‑Rank Adaptation (LoRA). By the end you will have a peft‑enabled checkpoint that can be loaded with a single GPU (8 GB VRAM minimum) and used for inference via the Hugging Face Transformers pipeline.

Prerequisites

• Python ≥ 3.9, pip, and a virtual environment
• An NVIDIA GPU with at least 8 GB VRAM (CUDA 11.8 or newer)
• Basic familiarity with git and Hugging Face Hub
• A small text dataset (CSV or JSONL) for domain adaptation

#Step 1Set up the environment

Create a fresh virtual environment and install the required libraries.

python -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install torch==2.2.0 transformers==4.38.0 datasets==2.16.0 peft==0.7.0 accelerate==0.27.0

#Step 2Prepare the dataset

For this tutorial we use a CSV file data/domain_corpus.csv with two columns: instruction and output. Load it with datasets.

from datasets import load_dataset

data_files = {"train": "data/domain_corpus.csv"}
dataset = load_dataset("csv", data_files=data_files, split="train")

Optionally, inspect a few rows:

print(dataset[:3])

If your data is in JSONL, replace "csv" with "json" and adjust data_files accordingly.

#Step 3Tokenize the data

We will use the tokenizer that matches the base model. The tokenization function pads to the longest sequence in each batch and truncates to the model’s max length (2048 tokens for LLaMA‑7B).

from transformers import AutoTokenizer

model_name = "meta-llama/Llama-2-7b-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)

def tokenize_fn(example):
    # Concatenate instruction and output with a separator token
    prompt = f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"
    return tokenizer(prompt, truncation=True, max_length=2048)

tokenized_dataset = dataset.map(tokenize_fn, batched=False)

#Step 4Configure LoRA with PEFT

LoRA adds two trainable matrices (lora_A and lora_B) to each linear layer. The following config uses rank 8 and a modest learning rate.

from peft import LoraConfig, get_peft_model

lora_cfg = LoraConfig(
    r=8,                # rank
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],  # adapt only query/value projections
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

# Load the base model in 4‑bit quantized mode to fit on a single GPU
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained(
    model_name,
    load_in_4bit=True,
    device_map="auto",
    torch_dtype="auto",
)

model = get_peft_model(base_model, lora_cfg)

#Step 5Set up Accelerate for distributed training (single GPU)

Create an accelerate config that enables mixed‑precision (fp16) and gradient accumulation.

accelerate config

When prompted, choose:

• Compute environment: This machine
• GPU(s): 1
• Mixed precision: fp16
• Gradient accumulation steps: 4
• DeepSpeed: No

Save the config (default location ~/.cache/huggingface/accelerate/default_config.yaml).

#Step 6Train the LoRA adapters

We will train for 3 epochs with a batch size of 2 (effective batch size = 8 due to accumulation).

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="lora-finetuned",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    save_steps=200,
    save_total_limit=2,
    report_to="none",  # disable wandb/MLflow
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    tokenizer=tokenizer,
)

trainer.train()

After training, the LoRA weights are saved inside lora-finetuned.

#Step 7Merge LoRA adapters (optional)

If you prefer a single checkpoint without PEFT wrappers, merge the adapters into the base model.

from peft import PeftModel

# Load the saved adapter
adapter = PeftModel.from_pretrained(base_model, "lora-finetuned")
merged_model = adapter.merge_and_unload()
merged_model.save_pretrained("merged-llama7b")
tokenizer.save_pretrained("merged-llama7b")

The merged-llama7b directory can now be loaded like any regular Hugging Face model.

#Step 8Run inference

Test the fine‑tuned model with a simple prompt.

from transformers import pipeline

pipe = pipeline(
    "text-generation",
    model="merged-llama7b",
    tokenizer=tokenizer,
    torch_dtype="auto",
    device_map="auto",
)

prompt = "Explain the benefits of LoRA for low‑resource fine‑tuning."
output = pipe(prompt, max_new_tokens=150, do_sample=True, temperature=0.7)
print(output[0]["generated_text"])

You should see domain‑specific language reflecting the training data.

#Key takeaways

#Common pitfalls

#Next steps

• Parameter-efficient multi‑task adapters – combine LoRA with adapters for different tasks in the same model.
• LoRA hyper‑parameter sweep – experiment with r values (4, 16) and lora_alpha to balance quality vs. size.
• Deploy with vLLM or Text Generation Inference – serve the merged checkpoint at low latency.
• Share on the Hub – push both base model and LoRA adapter to a private repo for team collaboration.

#FAQ

What is the difference between LoRA and full‑model fine‑tuning?
LoRA adds low‑rank matrices to existing weights, keeping the original parameters frozen. This reduces memory and compute by orders of magnitude while preserving the base model’s knowledge.

Can I fine‑tune a model larger than 7 B on a single GPU?
Yes, if you combine 4‑bit quantization, gradient checkpointing, and LoRA. Models up to 13 B can fit on 16 GB GPUs with careful batch sizing.

Do I need to merge adapters before deployment?
Merging is optional. Keeping adapters separate reduces storage and allows you to swap them at runtime, but some serving stacks (e.g., vLLM) expect a single model file, making merging convenient.

flowchart LR
    A["Start: Prepare env"] --> B["Load dataset"]
    B --> C["Tokenize"]
    C --> D["Configure LoRA"]
    D --> E["Accelerate config"]
    E --> F["Train"]
    F --> G["Merge (optional)"]
    G --> H["Inference"]
    H --> I["Deploy / Share"]