PermitQA: A Benchmark for Retrieval Augmented Generation in Wind Siting and Permitting domain
In the rapidly evolving landscape of Natural Language Processing (NLP) and text generation, the emergence of Retrieval Augmented Generation (RAG) presents a promising avenue for improving the quality and reliability of generated text by leveraging information retrieved from user specified database....
Saved in:
Main Authors | , , , , , , , |
---|---|
Format | Journal Article |
Language | English |
Published |
21.08.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | In the rapidly evolving landscape of Natural Language Processing (NLP) and
text generation, the emergence of Retrieval Augmented Generation (RAG) presents
a promising avenue for improving the quality and reliability of generated text
by leveraging information retrieved from user specified database. Benchmarking
is essential to evaluate and compare the performance of the different RAG
configurations in terms of retriever and generator, providing insights into
their effectiveness, scalability, and suitability for the specific domain and
applications. In this paper, we present a comprehensive framework to generate a
domain relevant RAG benchmark. Our framework is based on automatic
question-answer generation with Human (domain experts)-AI Large Language Model
(LLM) teaming. As a case study, we demonstrate the framework by introducing
PermitQA, a first-of-its-kind benchmark on the wind siting and permitting
domain which comprises of multiple scientific documents/reports related to
environmental impact of wind energy projects. Our framework systematically
evaluates RAG performance using diverse metrics and multiple question types
with varying complexity level. We also demonstrate the performance of different
models on our benchmark. |
---|---|
DOI: | 10.48550/arxiv.2408.11800 |