|国家预印本平台
首页|PLoRA: Efficient LoRA Hyperparameter Tuning for Large Models

PLoRA: Efficient LoRA Hyperparameter Tuning for Large Models

PLoRA: Efficient LoRA Hyperparameter Tuning for Large Models

来源:Arxiv_logoArxiv
英文摘要

Low-rank Adaptation (LoRA) has gained popularity as a fine-tuning approach for Large Language Models (LLMs) due to its low resource requirements and good performance. While a plethora of work has investigated improving LoRA serving efficiency by serving multiple LoRAs concurrently, existing methods assume that a wide range of LoRA adapters are available for serving. In our work, we conduct extensive empirical studies to identify that current training paradigms do not utilize hardware resources efficiently and require high overhead to obtain a performant LoRA. Leveraging these insights, we propose PLoRA, which automatically orchestrates concurrent LoRA fine-tuning jobs under given hardware and model constraints and develops performant kernels to improve training efficiency. Our experimental studies show that PLoRA reduces the makespan of LoRA fine-tuning over a given hyperparameter search space by up to 7.52x and improves training throughput by up to 12.8x across a range of state-of-the-art LLMs.

Minghao Yan、Zhuang Wang、Zhen Jia、Shivaram Venkataraman、Yida Wang

计算技术、计算机技术自动化技术、自动化技术设备

Minghao Yan,Zhuang Wang,Zhen Jia,Shivaram Venkataraman,Yida Wang.PLoRA: Efficient LoRA Hyperparameter Tuning for Large Models[EB/OL].(2025-08-04)[2025-08-23].https://arxiv.org/abs/2508.02932.点此复制

评论