editor@irjsrr.com
+91 9634765329
e-ISSN: 3108-1711
logo

International Research Journal of Scientific Reports and Reviews

LoginTrack
Published

Energy-Efficient Training of Large Language Models Through Sparse Attention and Low-Rank Adaptation (LoRA-S)

Anjani Kumar Tiwari
,
Pawar Harish
doi_icon
10.66542/irjsrr.v2.i1.000022

Published in January-June 2026 (Vol. 2, Issue 1, 2026)

Energy-Efficient Training of Large Language Models Through Sparse Attention and Low-Rank Adaptation (LoRA-S) - Issue cover

Keywords

Large Language ModelsLow-Rank Adaptation (LoRA)Sparse AttentionEnergy-Efficient TrainingGreen

Abstract

Abstract: Large language models (LLMs) such as GPT and BERT have revolutionized natural language processing but impose enormous computational and energy costs due to their massive parameter sizes and quadratic attention complexity. This study introduces LoRA-S, a unified framework that combines Low-Rank Adaptation (LoRA) with sparse attention mechanisms to achieve energy-efficient and scaljournalable training of transformer-based LLMs. By freezing pretrained weights and injecting low-rank trainable matrices into attention and feed-forward layers, LoRA reduces the number of trainable parameters by over 90%, significantly lowering gradient computation and memory overhead. Simultaneously, sparse attention restricts token interactions to structured subsets, cutting attention-related FLOPs from 100 G to 7 G in WikiText-2 experiments. Comparative analysis across Full Fine-Tuning, LoRA, and LoRA-S demonstrates that LoRA-S achieves the lowest energy consumption of 22,380 J (6.22 Wh) while maintaining competitive task performance, with perplexity of 115.26 on WikiText-2 and sentiment classification accuracy of 73.90% on IMDB. Pareto frontier analysis confirms LoRA-S as an optimal trade-off between computational efficiency and predictive capability, enabling resource-constrained and eco-friendly model deployment. These results establish LoRA-S as a practical step toward Green AI, providing a novel, integrated approach to minimize FLOPs and parameter updates without substantially compromising LLM performance.

References

  1. [1]Hadi, M. U., Qureshi, R., Shah, A., Irfan, M., Zafar, A., Shaikh, M. B., ... & Mirjalili, S. (2023). Large language models: a comprehensive survey of their applications, challenges, limitations, and future prospects. Authorea preprints, 1(3), 1-26.
  2. [2]Raiaan, M. A. K., Mukta, M. S. H., Fatema, K., Fahad, N. M., Sakib, S., Mim, M. M. J., ... & Azam, S. (2024). A review on large language models: Architectures, applications, taxonomies, open issues, and challenges. IEEE Access, 12, 26839-26874.
  3. [3]Jonnala, R., Yang, J., Lee, Y., Liang, G., & Cao, Z. (2025). Measuring and improving the efficiency of Python code generated by LLMs using cot prompting and fine-tuning. IEEE Access.
  4. [4]Mussa, A., Tuimebayev, Z., & Mansurova, M. (2025). Make Large Language Models Efficient: A Review. IEEE Access.DOI: 10.1109/ACCESS.2025.3605110
  5. [5]Wang, L., Chen, S., Jiang, L., Pan, S., Cai, R., Yang, S., & Yang, F. (2025). Parameter-efficient fine-tuning in large language models: a survey of methodologies. Artificial Intelligence Review, 58(8), 227.https://doi.org/10.1007/s10462-025-11236-4
  6. [6]Yuan, Z., Sun, W., Liu, Y., Zhou, H., Zhou, R., Li, Y., ... & Ye, Y. (2025). EfficientLLM: Efficiency in Large Language Models. arXiv preprint arXiv:2505.13840.https://doi.org/10.48550/arXiv.2505.13840
  7. [7]Usman, Y., Ihejirika, C. J., Offor, S. N., Robert, A., & Chataut, R. (2025). Green cybersecurity: leveraging AI, ML, and LLMs to optimize energy, threat detection, and sustainability Frameworks. IEEE Access.DOI: 10.1109/ACCESS.2025.3602451
  8. [8]Shahzad, T., Mazhar, T., Tariq, M. U., Ahmad, W., Ouahada, K., & Hamam, H. (2025). A comprehensive review of large language models: issues and solutions in learning environments. Discover Sustainability, 6(1), 27.https://doi.org/10.1007/s43621-025-00815-8
  9. [9]Wu, Y., Kan, S., Zeng, M., & Li, M. (2023, August). Singularformer: Learning to Decompose Self-Attention to Linearize the Complexity of Transformer. In IJCAI (pp. 4433-4441).
  10. [10]Sarpietro, R. E., Pino, C., Coffa, S., Messina, A., Palazzo, S., Battiato, S., ... & Rundo, F. (2022). Explainable deep learning system for advanced silicon and silicon carbide electrical wafer defect map assessment. IEEE Access, 10, 99102-99128.DOI: 10.1109/ACCESS.2022.3204278
  11. [11]Yin, D., Zhao, T. F., Fan, D. P., Li, S., Du, B., Sun, X., & Hu, S. M. (2025). Remote sensing tuning: A survey. Computational Visual Media.
  12. [12]Sharma, S. (2024). Generalization and Fine-Tuning of Robotic Foundation Models.
  13. [13]Taylor, N., Ghose, U., Rohanian, O., Nouriborji, M., Kormilitzin, A., Clifton, D. A., & Nevado-Holgado, A. (2024). Efficiency at scale: investigating the performance of diminutive language models in clinical tasks. Artificial intelligence in medicine, 157, 103002.https://doi.org/10.1016/j.artmed.2024.103002
  14. [14]Nwaiwu, S. (2025). Parameter-efficient fine-tuning for low-resource text classification: a comparative study of LoRA, IA3, and ReFT. Frontiers in Big Data, 8, 1677331.https://doi.org/10.3389/fdata.2025.1677331
  15. [15]Ayyat, M., Osman, M., & Nadeem, T. (2025). Opportunities and challenges of foundation models in industrial manufacturing. IEEE Access.
  16. [16]Kumar, P. (2024). Large language models (LLMs): survey, technical frameworks, and future challenges. Artificial Intelligence Review, 57(10), 260.https://doi.org/10.1007/s10462-024-10888-y
  17. [17]Tu, X., He, Z., Huang, Y., Zhang, Z. H., Yang, M., & Zhao, J. (2024). An overview of large AI models and their applications. Visual Intelligence, 2(1), 34.https://doi.org/10.1007/s44267-024-00065-8
  18. [18]Fakhabi, M. M., Hamidian, S. M., & Aliehyaei, M. (2024). Exploring the role of the Internet of Things in green buildings. Energy Science & Engineering, 12(9), 3779-3822.DOI: 10.1002 /ese3.1840
  19. [19]Barbierato, E., & Gatti, A. (2024). Toward green AI: A methodological survey of the scientific literature. IEEE Access, 12, 23989-24013.
  20. [20]Cong, S., & Zhou, Y. (2023). A review of convolutional neural network architectures and their optimizations. Artificial Intelligence Review, 56(3), 1905-1969.https://doi.org/10.1007/s10462-022-10213-5
  21. [21]Ahmed, S. F., Alam, M. S. B., Hassan, M., Rozbu, M. R., Ishtiak, T., Rafa, N., ... & Gandomi, A. H. (2023). Deep learning modelling techniques: current progress, applications, advantages, and challenges. Artificial Intelligence Review, 56(11), 13521-13617.https://doi.org/10.1007/s10462-023-10466-8

Authors (2)

Anjani Kumar Tiwari

Department of Civil Engineerin...

View all publications →

Pawar Harish

Department of Electronics & Co...

View all publications →

Download Article

PDF

Best for printing and citation

File size: 1.0 MB
Format: PDF

Download Article

PDF

Best for printing and citation

File size: 1.0 MB
Format: PDF

Article Information

Published in:

January-June 2026 (Vol. 2, Issue 1, 2026)

IRJSRR120022

IRJSRR-01-000022

19-42

2026-05-19

Article Impact

Views:4,682
Downloads:1,186
scite_
PlumX Metrics Badge

How to Cite

Kumar, A., & Harish (2026). Energy-Efficient Training of Large Language Models Through Sparse Attention and Low-Rank Adaptation (LoRA-S). International Research Journal of Scientific Reports and Reviews, 2(1), 19-42. DOI:https://doi.org/10.66542/irjsrr.v2.i1.000022

Article Actions

Whatsapp