# Fine‑Tune a 7B LLM with LoRA on Hugging Face in One GPU

> Learn how to fine‑tune a 7B LLM using LoRA and Hugging Face PEFT on a single GPU. Step‑by‑step code, pitfalls, and next steps for domain‑specific models.

- URL: https://ferdousazad.com/blog/fine-tune-llm-lora-huggingface-one-gpu
- Author: Ferdous Azad (Founder @ DataXgen AI)
- Published: 2026-11-08
- Updated: 2026-11-08
- Tags: LLM, LoRA, Hugging Face, PEFT, fine‑tuning, AI

## Intro  

In this tutorial we will fine‑tune a 7‑billion‑parameter LLaMA‑style model on a custom text dataset using Low‑Rank Adaptation (LoRA). By the end you will have a `peft`‑enabled checkpoint that can be loaded with a single GPU (8 GB VRAM minimum) and used for inference via the Hugging Face Transformers pipeline.  

**Prerequisites**  

- Python ≥ 3.9, pip, and a virtual environment  
- An NVIDIA GPU with at least 8 GB VRAM (CUDA 11.8 or newer)  
- Basic familiarity with `git` and Hugging Face Hub  
- A small text dataset (CSV or JSONL) for domain adaptation  

> [!NOTE]  
> The steps work for any small LLM that supports `torch_dtype=torch.float16`. Replace the model ID if you prefer a different base (e.g., `mistralai/Mistral-7B-v0.1`).  

## Step 1: Set up the environment  

Create a fresh virtual environment and install the required libraries.

```bash
python -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install torch==2.2.0 transformers==4.38.0 datasets==2.16.0 peft==0.7.0 accelerate==0.27.0
```

> [!TIP]  
> If you have a newer GPU (e.g., RTX 4090) you can install the latest `torch` wheel directly from the PyPI index with CUDA support.  

## Step 2: Prepare the dataset  

For this tutorial we use a CSV file `data/domain_corpus.csv` with two columns: `instruction` and `output`. Load it with `datasets`.

```python
from datasets import load_dataset

data_files = {"train": "data/domain_corpus.csv"}
dataset = load_dataset("csv", data_files=data_files, split="train")
```

Optionally, inspect a few rows:

```python
print(dataset[:3])
```

If your data is in JSONL, replace `"csv"` with `"json"` and adjust `data_files` accordingly.  

## Step 3: Tokenize the data  

We will use the tokenizer that matches the base model. The tokenization function pads to the longest sequence in each batch and truncates to the model’s max length (2048 tokens for LLaMA‑7B).

```python
from transformers import AutoTokenizer

model_name = "meta-llama/Llama-2-7b-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)

def tokenize_fn(example):
    # Concatenate instruction and output with a separator token
    prompt = f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"
    return tokenizer(prompt, truncation=True, max_length=2048)

tokenized_dataset = dataset.map(tokenize_fn, batched=False)
```

## Step 4: Configure LoRA with PEFT  

LoRA adds two trainable matrices (`lora_A` and `lora_B`) to each linear layer. The following config uses rank 8 and a modest learning rate.

```python
from peft import LoraConfig, get_peft_model

lora_cfg = LoraConfig(
    r=8,                # rank
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],  # adapt only query/value projections
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

# Load the base model in 4‑bit quantized mode to fit on a single GPU
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained(
    model_name,
    load_in_4bit=True,
    device_map="auto",
    torch_dtype="auto",
)

model = get_peft_model(base_model, lora_cfg)
```

> [!IMPORTANT]  
> `load_in_4bit=True` requires the `bitsandbytes` library, which is automatically installed with the latest `torch` wheel.  

## Step 5: Set up Accelerate for distributed training (single GPU)  

Create an `accelerate` config that enables mixed‑precision (`fp16`) and gradient accumulation.

```bash
accelerate config
```

When prompted, choose:

- **Compute environment**: `This machine`
- **GPU(s)**: `1`
- **Mixed precision**: `fp16`
- **Gradient accumulation steps**: `4`
- **DeepSpeed**: `No`

Save the config (default location `~/.cache/huggingface/accelerate/default_config.yaml`).  

## Step 6: Train the LoRA adapters  

We will train for 3 epochs with a batch size of 2 (effective batch size = 8 due to accumulation).  

```python
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="lora-finetuned",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    save_steps=200,
    save_total_limit=2,
    report_to="none",  # disable wandb/MLflow
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    tokenizer=tokenizer,
)

trainer.train()
```

After training, the LoRA weights are saved inside `lora-finetuned`.  

## Step 7: Merge LoRA adapters (optional)  

If you prefer a single checkpoint without PEFT wrappers, merge the adapters into the base model.

```python
from peft import PeftModel

# Load the saved adapter
adapter = PeftModel.from_pretrained(base_model, "lora-finetuned")
merged_model = adapter.merge_and_unload()
merged_model.save_pretrained("merged-llama7b")
tokenizer.save_pretrained("merged-llama7b")
```

The `merged-llama7b` directory can now be loaded like any regular Hugging Face model.  

## Step 8: Run inference  

Test the fine‑tuned model with a simple prompt.

```python
from transformers import pipeline

pipe = pipeline(
    "text-generation",
    model="merged-llama7b",
    tokenizer=tokenizer,
    torch_dtype="auto",
    device_map="auto",
)

prompt = "Explain the benefits of LoRA for low‑resource fine‑tuning."
output = pipe(prompt, max_new_tokens=150, do_sample=True, temperature=0.7)
print(output[0]["generated_text"])
```

You should see domain‑specific language reflecting the training data.  

## Key takeaways  

- LoRA reduces GPU memory by injecting tiny trainable matrices, enabling 7 B‑parameter models on a single GPU.  
- Using `bitsandbytes` 4‑bit quantization plus `accelerate`’s mixed‑precision yields a memory footprint under 8 GB.  
- Only a few epochs on a modest dataset can imprint domain knowledge without catastrophic forgetting.  
- The PEFT workflow lets you keep the base model untouched and share adapters separately.  

## Common pitfalls  

| Symptom | Likely cause | Fix |
|---|---|---|
| OOM error even with 4‑bit | `bitsandbytes` not installed or CUDA mismatch | Reinstall `bitsandbytes` with matching CUDA version |
| Tokenizer truncates important context | `max_length` too low | Increase `max_length` to 4096 if model supports it |
| Training stalls at 0 % loss improvement | Learning rate too high/low | Try `1e-4` or `5e-5` and monitor loss |
| Generated text repeats the prompt | `do_sample=False` in inference | Enable sampling (`do_sample=True`) or adjust temperature |

## Next steps  

- **Parameter-efficient multi‑task adapters** – combine LoRA with adapters for different tasks in the same model.  
- **LoRA hyper‑parameter sweep** – experiment with `r` values (4, 16) and `lora_alpha` to balance quality vs. size.  
- **Deploy with vLLM or Text Generation Inference** – serve the merged checkpoint at low latency.  
- **Share on the Hub** – push both base model and LoRA adapter to a private repo for team collaboration.  

## FAQ  

**What is the difference between LoRA and full‑model fine‑tuning?**  
LoRA adds low‑rank matrices to existing weights, keeping the original parameters frozen. This reduces memory and compute by orders of magnitude while preserving the base model’s knowledge.

**Can I fine‑tune a model larger than 7 B on a single GPU?**  
Yes, if you combine 4‑bit quantization, gradient checkpointing, and LoRA. Models up to 13 B can fit on 16 GB GPUs with careful batch sizing.

**Do I need to merge adapters before deployment?**  
Merging is optional. Keeping adapters separate reduces storage and allows you to swap them at runtime, but some serving stacks (e.g., vLLM) expect a single model file, making merging convenient.  

---  

```mermaid
flowchart LR
    A["Start: Prepare env"] --> B["Load dataset"]
    B --> C["Tokenize"]
    C --> D["Configure LoRA"]
    D --> E["Accelerate config"]
    E --> F["Train"]
    F --> G["Merge (optional)"]
    G --> H["Inference"]
    H --> I["Deploy / Share"]
```