Implementasi Retrieval-Augmented Generation (Rag) Untuk Fact-Checking Otomatis Pada Klaim Kesehatan

Abstract

The rapid growth of digital platforms has accelerated public access to online health information, but it has also become a primary channel for the spread of health misinformation. Large Language Models (LLMs) have been widely adopted to address this problem through pure generative approaches; however, they frequently produce convincing yet inaccurate information, a phenomenon known as hallucination. This study implements a Retrieval-Augmented Generation approach for automatic fact-checking of health claims using the PubHealth dataset, and compares its performance against Corrective RAG (CRAG), a retrieval-evaluation-based RAG variant that filters retrieved evidence into Correct, Incorrect, or Ambiguous actions before answer generation. The proposed pipeline combines a FAISS-based semantic retriever, CrossEncoder re-ranking, and a FLAN-T5 generator to classify each claim as Fact or Myth while supplying relevant supporting evidence, and is integrated into a Gradio-based interface for claim verification and dataset expansion. The models were evaluated using accuracy, precision, recall, F1-Score, and Cohen's Kappa on the 3,770 yes/no questions that receive an explicit Fact/Myth verdict, and using faithfulness, context precision, and answer relevancy on the full 6,704-item question dataset. The FAISS retriever, CrossEncoder re-ranker, and FLAN-T5 generator were further tested through a leave-one-out ablation study combined with McNemar's test on the complete set of 3,770 yes/no questions. The results show that the RAG model outperformed CRAG on the core classification metrics, achieving an accuracy of 0.564, an F1-score of 0.564, and a Cohen's Kappa of 0.128, compared to CRAG's accuracy of 0.493, F1-score of 0.492, and a near-chance Cohen's Kappa of -0.010, whereas CRAG achieved a higher context precision (0.452 versus 0.333) and answer relevancy (0.731 versus 0.157). The ablation study and McNemar's test confirmed that the FAISS retriever and the FLAN-T5 generator each contribute a statistically significant improvement to classification accuracy (p < 0.05).