LoRA and QLoRA recommendations for LLMs

This page gives you configuration recommendations for tuning large language models (LLM) on Vertex AI by using Low-Rank Adaptation of Large Language Models (LoRA) and its more memory-efficient version, QLoRA.

Tuning recommendations

The following table summarizes our recommendations for tuning LLMs by using LoRA or QLoRA:

Specification Recommended Details
GPU memory efficiency QLoRA QLoRA has about 75% smaller peak GPU memory usage compared to LoRA.
Speed LoRA LoRA is about 66% faster than QLoRA in terms of tuning speed.
Cost efficiency LoRA While both methods are relatively inexpensive, LoRA is up to 40% less expensive than QLoRA.
Higher max sequence length QLoRA Higher max sequence length increases GPU memory consumption. QLoRA uses less GPU memory so it can support higher max sequence lengths.
Accuracy improvement Same Both methods offer similar accuracy improvements.
Higher batch size QLoRA QLoRA supports much higher batch sizes. For example, the following are batch size recommendations for tuning openLLaMA-7B on the following GPUs:
  • 1 x A100 40G:
    • LoRA: Batch size of 2 is recommended.
    • QLoRA: Batch size of 24 is recommended.
  • 1 x L4:
    • LoRA: Batch size of 1 fails with an out of memory error (OOM).
    • QLoRA: Batch size of 12 is recommended.
  • 1 x V100:
    • LoRA: Batch size of 1 fails with an out of memory error (OOM).
    • QLoRA: Batch size of 8 is recommended.