Which Layers Should LoRA Target? Picking Target Modules

// In plain English

LoRA (low-rank adaptation) fine-tunes a giant model by freezing its original weights and training a tiny pair of extra matrices alongside them. But here's the part most tutorials skip over: LoRA does not get applied to the whole model. It only attaches its little trainable adapters to the specific layers you point it at. That setting is called target modules — in code, the target_modules argument.

Think of a transformer as a long building with dozens of identical floors, and each floor has a handful of named rooms — the query room, the key room, the value room, the output room, and a couple of big MLP rooms. LoRA is like hiring a renovation crew that can only work in the rooms you list on the work order. List two rooms per floor and the job is cheap and fast. List every room and you get a deeper renovation, but it costs more time and memory. The rooms you don't list stay exactly as they were.

So target_modules is the work order. It answers a single question: which of the model's linear layers get a LoRA adapter, and which are left untouched? Beginners usually copy a default list from a blog post without knowing what it controls — which is exactly the knob this article opens up.

// Why it matters

Target modules is the quiet knob that decides three things at once: how much memory training takes, how many parameters you train, and — crucially — what the adapter is even capable of learning. Two people can run "the same" LoRA fine-tune on the same data and get very different results purely because one targeted attention layers only and the other targeted everything.

• It sets the parameter budget. Every module you add multiplies the number of trainable adapter weights. Attention-only might train ~0.1% of the model; all-linear-layers can be several times that. More trainable parameters means more GPU memory for optimizer states and gradients.
• It bounds what can be learned. An adapter can only reshape the layers it's attached to. If a task needs the model to store or recall new facts, the MLP layers matter a lot — and skipping them leaves quality on the table no matter how high you set the rank.
• It's a silent failure mode. If you fat-finger a module name, PEFT may attach zero adapters and happily train nothing, or attach far fewer than you expected. The run completes, the loss barely moves, and nothing errors out. Knowing the right names saves hours of confusion.

This is distinct from rank and alpha. Rank controls how expressive each adapter is; target modules control how many adapters there are and where they sit. You tune them together, but they answer different questions — and getting target modules wrong can't be rescued by cranking rank.

// How it works

Inside one transformer block there are two big sub-blocks: self-attention and the MLP (also called the feed-forward network). Each is built from named linear projections, and those names are what you put in target_modules. The exact strings vary by model family, but the Llama-style naming is the one you'll see most often:

LoRA wraps each targeted module: the frozen original weight W stays put, and a trainable low-rank pair B·A is added in parallel. At every targeted layer the output becomes W·x + (B·A)·x. Modules you don't target keep just W·x — no adapter, nothing to train.

The historical default, going back to the original LoRA paper, was to target only q_proj and v_proj — the query and value projections. The authors found that adapting just those two gave most of the benefit for the least cost on the tasks they tested. For years, copy-pasted configs carried that default forward (the accent boxes above).

The now-common practice is the opposite: target all linear layers, attention and MLP. The QLoRA work popularized this — applying LoRA to every linear projection closed the gap with full fine-tuning more reliably than the query/value-only recipe. Modern tooling even gives you a shortcut for it:

from peft import LoraConfig

# Historical default: attention query + value only. Cheapest.
cfg_minimal = LoraConfig(
    r=8, lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
)

# Common modern recipe: every attention + MLP projection.
cfg_all = LoraConfig(
    r=8, lora_alpha=16,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
)

# Shortcut: let PEFT find every Linear layer for you.
cfg_auto = LoraConfig(r=8, lora_alpha=16, target_modules="all-linear")

// Attention vs. MLP: what each one buys you

The single most useful mental model: attention layers move information around; MLP layers store and transform it. That split is why your choice of target modules changes what the adapter can learn, not just how big it is.

A rough rule of thumb: if your fine-tune is about behavior, tone, or following a format, attention-heavy targeting may be enough. If it's about teaching the model new domain content or knowledge, you almost certainly want the MLP layers in the list too — that's where research suggests most factual capacity sits. When in doubt, target all linear layers; it's the more robust default and the cost is usually worth it.

// Finding the right module names

The names q_proj, gate_proj, and so on are Llama/Mistral conventions, not universal. GPT-2-style models name their packed attention projection c_attn; some architectures fuse query/key/value into a single qkv_proj; others use query, key, value. If you target a name the model doesn't have, PEFT attaches no adapter there — silently.

Never guess. Print the model and read the actual module names before you write your config:

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("your/model-id")

# List every Linear module's name — these are valid target_modules.
for name, module in model.named_modules():
    if isinstance(module, torch.nn.Linear):
        print(name)
# e.g. model.layers.0.self_attn.q_proj
#      model.layers.0.mlp.gate_proj
# PEFT matches on the suffix, so "q_proj" hits all layers at once.

PEFT matches each entry against the end of every module's full name, so a single short string like q_proj applies the adapter to that projection in every transformer block at once. After building your adapted model, always sanity-check that you trained what you intended:

from peft import get_peft_model

peft_model = get_peft_model(model, cfg_all)

# Prints trainable params and the % of the model they represent.
# If this is ~0% or far lower than expected, your names are wrong.
peft_model.print_trainable_parameters()

// Going deeper

Once the basics click, a few finer points separate a working config from a well-tuned one.

Target modules and rank interact. Spreading the same parameter budget across more modules at low rank often beats piling rank onto just two modules. If you switch from query/value-only to all-linear, you can usually lower the rank and keep a similar total parameter count. They're separate knobs, but you tune them as a pair.

QLoRA changes the calculus, not the choice. With QLoRA the base model is quantized to 4-bit, which slashes its memory footprint — and the adapters stay in higher precision. Because the frozen weights cost so little now, targeting all linear layers is cheap enough that it became the standard QLoRA recipe. The module choice is the same; the memory headroom is different.

Some layers are usually left out on purpose. The output/lm_head and token-embedding layers are typically not LoRA-targeted by default. Adapting them can help in narrow cases (for example, teaching a model new tokens), but it adds a lot of parameters and risks destabilizing training, so it's an advanced, deliberate move — not part of all-linear.

It's an empirical decision, so evaluate. There is no module list that's optimal for every task. The honest workflow is to start from a sensible default (all-linear, or query/value-only if you're memory-bound), then evaluate the fine-tuned model on a held-out set and compare. Treat target modules as something you test, not something you copy and trust. For the bigger picture of how LoRA fits among other approaches, see full fine-tuning vs. PEFT.

The durable lesson: target_modules is not boilerplate. It's the line that decides which parts of the model your adapter can reshape — and for any fine-tune that needs to learn real content, leaving the MLP layers out is the most common way to quietly leave quality on the table.