Why Fine-Tune a Coding LLM?

While general flagship models are excellent at general programming, they fail when faced with proprietary component libraries, specific company styling guidelines, or custom framework APIs. Under a traditional "vibe coding" setup, developers spend huge chunks of their context window feeding code style guides and templates back to the model.

Fine-tuning solves this. By updating the weights of an open-access model like Qwen 2.5-Coder (14B or 32B), you can bake your custom syntax directly into the model's neural paths.

In this guide, we will walk through a complete step-by-step pipeline using Unsloth and LoRA (Low-Rank Adaptation) to train Qwen 2.5-Coder to output styling code adhering strictly to a specific CSS design system.

---

1. Formatting the Training Dataset

Fine-tuning requires a clean dataset of instruction-response pairs. For code customization, this dataset must contain code snippets accompanied by the specification prompts that trigger them.

Create a JSON dataset file named vibe_coding_dataset.json following this structure:

json

[
  {
    "instruction": "Build a responsive product grid card using the standard theme classes.",
    "input": "",
    "output": "<div className=\"card card-interactive\">\n  <span className=\"badge badge-primary\">New</span>\n  <h4 className=\"card-title\">Product Name</h4>\n  <p className=\"card-desc\">Product description goes here.</p>\n</div>"
  },
  {
    "instruction": "Generate a secondary outline action button.",
    "input": "",
    "output": "<button className=\"btn btn-secondary btn-sm\">\n  Action Label\n</button>"
  }
]

Tip: Ensure you have at least 500 to 1,000 diverse prompt-code samples for high-quality instruction learning.

---

2. Writing the Unsloth Fine-Tuning Script

Unsloth is a popular, highly optimized library that speeds up LoRA fine-tuning by up to 2x and reduces memory footprint, enabling 14B models to be trained on a single RTX 3090/4090 GPU.

Here is the complete Python script to execute the training loop:

```python

train_qwen.py

from unsloth import FastLanguageModel import torch from datasets import load_dataset from trl import SFTTrainer from transformers import TrainingArguments

max_seq_length = 4096 # Supports up to 128K, 4K is plenty for standard components dtype = None # Auto-detection load_in_4bit = True # Reduces memory consumption by 75%

1. Load Pre-trained Qwen 2.5 Coder Model

model, tokenizer = FastLanguageModel.from_pretrained( model_name = "Qwen/Qwen2.5-Coder-14B-Instruct", max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, )

⚡ SPONSORED TUTORIAL ADStrategic Developer Ad Placement Slot (mid_article)Set NEXT_PUBLIC_ADSENSE_CLIENT_ID in .env to activate live ad delivery

2. Configure Parameter-Efficient LoRA Weights

model = FastLanguageModel.get_peft_model( model, r = 16, target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], lora_alpha = 16, lora_dropout = 0, bias = "none", use_gradient_checkpointing = "unsloth", random_state = 3407, )

3. Load & Process Dataset

prompt_format = """Below is an instruction that describes a coding task. Write a response that appropriately completes the request.

Instruction:

{}

Response:

{}"""

def format_prompts(fields): instructions = fields["instruction"] outputs = fields["output"] texts = [] for inst, out in zip(instructions, outputs): texts.append(prompt_format.format(inst, out)) return { "text" : texts }

dataset = load_dataset("json", data_files="vibe_coding_dataset.json", split="train") dataset = dataset.map(format_prompts, batched=True)

4. Initialize Trainer Configuration

trainer = SFTTrainer( model = model, tokenizer = tokenizer, train_dataset = dataset, dataset_text_field = "text", max_seq_length = max_seq_length, dataset_num_proc = 2, packing = False, # Speed up training for short sequences args = TrainingArguments( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 5, max_steps = 60, # Increase to 300+ steps for full convergence learning_rate = 2e-4, fp16 = not torch.cuda.is_bf16_supported(), bf16 = torch.cuda.is_bf16_supported(), logging_steps = 1, output_dir = "outputs", ), )

5. Execute Training Run

trainer.train()

6. Save Fine-Tuned Model and LoRA Weights

model.save_pretrained_merged("qwen25-custom-vibe-coder", tokenizer, save_method="merged_16bit") print("✅ Custom Qwen2.5 model successfully fine-tuned & saved!")

code

---

## 3. Integrating the Custom Model into Your Vibe Loop

After training, you can export the merged model to GGUF format for use in Ollama:

bash

Compile and load your fine-tuned model into Ollama

ollama create custom-vibe-coder -f Modelfile

code

Where the `Modelfile` looks like:

dockerfile FROM ./qwen25-custom-vibe-coder-q8_0.gguf PARAMETER temperature 0.2 PARAMETER top_p 0.95 SYSTEM "You are Custom Vibe Coder. You strictly output code matching our custom CSS tokens and grid layouts." ``

Now, when you trigger your vibe coding API endpoint, point your model target to custom-vibe-coder`. The generated code will immediately match your organization's custom patterns with zero contextual examples needed!

---

Next Steps

Read our performance benchmarks in Qwen 2.5-Coder: The Open-Source AI Champion. Check out prompt checklists in the Prompt Library. * See how we run interactive workspace missions in the Dashboard Missions.

🚀 VIBE CODING CODEX ACADEMY

Ready to Stop Typing Code & Start Directing AI Systems?

Join 5,000+ developers building real production software with AI. Learn the 7-Stage Vibe Coding OS, harness engineering, and earn your verified Build DNA Certificate.

Enroll in Academy — Get Started Free →Try Harness Generator 🐎

⚡ SPONSORED ADVERTISEMENTStrategic Developer Ad Placement Slot (bottom_banner)Set NEXT_PUBLIC_ADSENSE_CLIENT_ID in .env to activate live ad delivery

Fine-Tuning Qwen 2.5 for Domain-Specific Vibe Coding Workflows