TELLM: Advancements in Knowledge Incorporation and Task-specific Enhancements of Large Language Models

• Published in: Iranian Conference on Electrical Engineering pp. 1 - 5
• Main Authors: Feizi, Fatemeh, HosseinNia, Amirhossein, Hemmatyar, MohammadMahdi, Rahimi, Fatemeh, Kaleibar, Farhoud Jafari
• Format: Conference Proceeding
• Language: English
• Published: IEEE 14.05.2024
• Subjects: Customer services; Deep Learning; Knowledge based systems; Large language models; Measurement; Mission critical systems; Question-answering; Reinforcement learning; Subject matter experts; Telecom
• ISSN: 2642-9527
• DOI: 10.1109/ICEE63041.2024.10667786

Customer service is crucial for any business to maintain good customer relationships and growth. However, addressing a wide variety of customer inquiries often requires deep domain expertise that may not be readily available. This paper presents TELLM, a customer service AI agent leveraging large language models to provide technical support for telecom companies. TELLM is trained using a knowledge base containing categorized technical solutions through dual-phase fine-tuning. It is further refined through reinforcement learning with feedback from subject matter experts. Evaluation on a real-world customer service dataset demonstrates TELLM outperforms prior approaches on automatic metrics and achieves high scores in human evaluation for response accuracy, clarity and effectiveness.