Analisis Perbandingan Kualitas Jawaban Pada Qa Berbasis Rag : Kombinasi Zero-Shot Instruction Prompting Dan Self-Verification

Abstract

Retrieval-Augmented Generation (RAG) reduces hallucinations by grounding language model (LM) responses in the context of retrieved results, but does not automatically guarantee evidence-based (faithful) responses. This study examines the effect of combining zero-shot instruction prompting and self-verification on the faithfulness of responses to answerable questions, across four RAG pipeline configurations: (1) RAG, (2) RAG + prompting, (3) RAG + self-verification, and (4) RAG + prompting + self-verification, which were tested on three scales of the Qwen3 language model (0.6B, 4B, 8B) using the SQuAD v2.0 dataset. A total of 100 queries were selected via stratified random sampling from the validation split to avoid topic bias. Faithfulness was measured at the claim level using an NLI model (DeBERTa-v3-large). The results show that basic RAG (configuration 1) achieved the highest average faithfulness score, while configurations 2–4 (with prompting and/or self-verification) showed relatively similar and lower scores. In SLM, adding self-verification lowered faithfulness the most, while in LLM the score remained relatively stable across configurations. A retrieval-quality control analysis indicated that this decline was linked to the generator's capacity rather than retrieval quality. These findings suggest that the combination of prompting and self-verification does not automatically improve the quality of evidence-based answers, and its benefits depend on the adequacy of the language model's capacity.