Evaluation of Arabic Named Entity Recognition Models on Sahih Al-Bukhari Text

Abstract

In this paper, the following four Arabic named entity recognition (ANER) models were applied to the Sahih Al-Bukhari (صحيح البخاري) dataset: CAMeLBERT-CA Hatmimoha, Marefa-NER, and Stanza. This study's main aim is to identify the best-performing model for use with other Hadith datasets. The Stanza and Marefa-NER models are best because they obtained F1-scores of 0.826191 and 0.807396, respectively. Then, a new test dataset of approximately 5,000 words was created based on the CANERCorpus annotation. The four models were evaluated using the latest test dataset and had disappointing F1-scores, although Hatmimoha had the best results. This problem likely arose as a result of the small dataset. However, we observed that since the model has many named entity classes and matches the CANERCorpus labels, it could obtain a high performance, as the Hatmimoha and Marefa-NER models did.