Finetuning NepBerta for Nepali NER

Background

Natural Language Processing (NLP) and LLMs have gained significant traction in recent years, helping machines understand and interpret human languages more effectively. However, low-resource languages like Nepali still lag behind in data, models, resources, research, and even cultural factors. One key challenge is the lack of a culture that promotes contributing to public datasets and tools for free, which hinders the development of NLP models for such languages. To bridge this gap, we developed Finetuned NepBertA-NER, a Nepali Named Entity Recognition (NER) model based on NepBERTa.

Named Entity Recognition (NER) is a key NLP task that involves identifying and classifying entities (like people, organizations, and locations) in text. In widely spoken languages like English, state-of-the-art models for NER exist, but Nepali—spoken by millions—lacks similar resources.

This gap inspired the creation of NepBERTa, a BERT-based model designed for the Nepali language. Building on that, the Finetuned NepBertA-NER model was trained to recognize specific entities in Nepali, making it an important tool for extracting key information from Nepali news articles, analyzing customer feedback or social media posts in Nepali, and processing legal and financial documents in Nepali.

How the Model Was Fine-Tuned

The fine-tuning process involved adapting the pre-trained NepBERTa model to the task of Named Entity Recognition. Dataset Preparation: A custom dataset was annotated with labels like B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, and O, split into training and validation sets using an 80-20 ratio.

Model Setup: We used Hugging Face's AutoModelForTokenClassification to load the NepBERTa model and fine-tune it for the NER task, configured to identify 7 entity classes.

Training: The training was conducted using the AdamW optimizer with a learning rate of 5e-5. The model was trained for 5 epochs on a GPU. During training, batches of text were fed into the model to predict entity labels, with loss calculated based on the model's predictions and true labels.

Validation: After each epoch, the model's performance was validated using a validation set. The evaluation metrics included precision, recall, and F1-score.

Model Capabilities

The Finetuned NepBertA-NER model is capable of recognizing and classifying the following types of entities in Nepali text: Persons (PER) – Names of people, Organizations (ORG) – Names of companies, institutions, and other entities, and Locations (LOC) – Names of places, such as cities, countries, and landmarks.

For example, when given the sentence "काठमाडौं उपत्यकाको ऐतिहासिक पशुपतिनाथ मन्दिर धेरै प्रसिद्ध छ।", the model outputs: पशुपतिनाथ → Location (LOC) and काठमाडौं → Location (LOC).

This enables the model to assist in a range of NLP applications, including document parsing, information extraction, and sentiment analysis in the Nepali language.

How to Use the Model

You can easily use Finetuned NepBertA-NER through Hugging Face's transformers library. Load the pre-trained model and tokenizer using AutoModelForTokenClassification.from_pretrained("SynapseHQ/Finetuned-NER-NepBertA").

For large texts, it's good practice to process them in chunks using a predict_ner_chunked function that splits text, runs predictions, and collects entity tags (B-PER, I-PER, B-ORG, B-LOC).

Feed Nepali text into the function and get the entities. For example, the input "सङ्घीय लोकतान्त्रिक गणतन्त्र नेपालको प्रधानमन्त्री शेरबहादुर देउवा हुन्।" produces: [('शेरबहादुर', 'B-PER'), ('देउवा', 'I-PER'), ('नेपाल', 'B-LOC')].

Future Work

While the model performs well, there is always room for improvement. Larger, More Diverse Dataset: Expanding the dataset to include more varied domains—such as medical, legal, and financial texts—could improve versatility.

Addition of More Entity Types: Adding more entity types (such as dates, monetary values, and geopolitical entities) would enhance utility in a wider range of applications.

Cross-Lingual Training: Incorporating training data from similar languages like Hindi could make the model more robust. Leveraging More Advanced Architectures: Exploring newer transformer architectures like DeBERTa or RoBERTa could potentially yield better results.

Conclusion

The Finetuned-NepBertA-NER model is a step forward in enhancing NLP capabilities for the Nepali language. By providing a robust method to recognize and categorize named entities, it opens doors to various applications—from text analysis and information extraction to sentiment analysis and legal document processing.

The model is available on Hugging Face: Finetuned NepBertA-NER. We encourage researchers, developers, and enthusiasts to explore and use this model in their projects.