Sparse Attention

Abstract: Large language models (LLMs) such as GPT and BERT have revolutionized natural language processing but impose enormous computational and energy costs due to their massive parameter sizes and quadratic attention complexity. This study introduces LoRA-S, a unified framework that combines Low-Rank Adaptation (LoRA) with sparse attention mechanisms to achieve energy-efficient and scaljournalable training of transformer-based LLMs. By freezing pretrained weights and injecting low-rank trainable matrices into attention and feed-forward layers, LoRA reduces the number of trainable parameters by over 90%, significantly lowering gradient computation and memory overhead. Simultaneously, sparse attention restricts token interactions to structured subsets, cutting attention-related FLOPs from 100 G to 7 G in WikiText-2 experiments. Comparative analysis across Full Fine-Tuning, LoRA, and LoRA-S demonstrates that LoRA-S achieves the lowest energy consumption of 22,380 J (6.22 Wh) while maintaining competitive task performance, with perplexity of 115.26 on WikiText-2 and sentiment classification accuracy of 73.90% on IMDB. Pareto frontier analysis confirms LoRA-S as an optimal trade-off between computational efficiency and predictive capability, enabling resource-constrained and eco-friendly model deployment. These results establish LoRA-S as a practical step toward Green AI, providing a novel, integrated approach to minimize FLOPs and parameter updates without substantially compromising LLM performance.