What We Do
The recent emergence of multilingual large language models (LLMs) is revolutionizing natural language processing, bridging communication gaps across diverse cultures and languages. However, to truly harness the potential of these models, it’s crucial to understand their strengths and limitations across a wide range of languages and tasks. MELT is designed with this in mind, offering a comprehensive approach to evaluate LLMs in various linguistic contexts. Recognizing that proficiency in one language or task does not guarantee similar performance elsewhere, MELT enables users to pinpoint specific areas for improvement, fostering the development of robust and reliable multilingual language technologies.
MELT includes ten carefully selected evaluation scenarios, each targeting a key aspect of LLM capability:
- Summarization: Evaluates the model’s ability to condense large texts while retaining essential information.
- Question-Answering: Assesses comprehension and accurate extraction of answers from provided contexts.
- Knowledge: Tests the model’s ability to recall and apply information across different domains.
- Sentiment Analysis: Measures the ability to detect and classify emotional tones in text.
- Text Classification: Evaluates accuracy in categorizing text into predefined labels.
- Toxic Detection: Identifies the model’s capacity to flag harmful or biased language.
- Language Modeling: Tests fluency and coherence in generating contextually appropriate text.
- Reasoning: Measures logical deduction and understanding of complex relationships.
- Math: Assesses competency in solving mathematical problems in text form.
- Information Retrieval: Tests effectiveness in searching, retrieving, and synthesizing relevant information.
MELT also includes tools to ensure the ethical deployment of LLMs:
- Bias Assessment: Identifies and mitigates potential biases in model outputs.
- Toxicity Assessment: Measures and controls the generation of harmful or offensive language.
- Fairness Evaluation: Ensures equitable performance across demographic groups and languages.
- Robustness Analysis: Examines resilience to noisy inputs and adversarial attacks, ensuring reliable performance in real-world scenarios.
MELT offers a holistic evaluation framework that not only assesses performance but also emphasizes ethical considerations, making it an essential tool for developing multilingual language models that are both effective and responsible. MELT currently supports the following languages and tasks:
| Model |
|---|
|
| Task | Dataset | Metric |
|---|---|---|
| Question Answering | XQuaD | Exact Match, F1 Score |
| MLQA | ||
| Text Summarization | VietNews | ROUGE-1, ROUGE-2, ROUGE-L, SummaC, BERTScore, Coverage, Density, Compression |
| WikiLingua | ||
| Sentiment Analysis | VLSP 2016 | Accuracy, F1 Score, AUC ROC, Expected Calibration Error at top-10, Accuracy at 10% coverage |
| UiT-VSFC | ||
| Text Classification | UiT-VSMEC | Accuracy, F1 Score, AUC ROC, Expected Calibration Error at top-10, Accuracy at 10% coverage |
| PhoATIS | ||
| Knowledge | ZaloE2E | Exact Match, F1 Score |
| ViMMRC | Accuracy, F1 Score, AUC ROC, Expected Calibration Error at top-10, Accuracy at 10% coverage |
|
| Toxicity Detection | UiT-ViCTSD | Accuracy, F1 Score, AUC ROC, Expected Calibration Error at top-10, Accuracy at 10% coverage |
| UiT-ViHSD | ||
| Information Retrieval | mMARCO | Mean Reciprocal Rank in top-10, Boosted Mean Reciprocal Rank in top-10, Normalized Discounted Cumulative Gain in top-10, Boosted Normalized Discounted Cumulative Gain in top-10 |
| mRobust04 | ||
| Language Modeling | MLQA-MLM | Exact Match, Character Error Rate, Word Error Rate, Character Edit Distance, Word Edit Distance, Perplexity |
| VSEC | ||
| Reasoning | Synthetic Reasoning - Natural Language |
Exact Match, F1 Score, Equivalent |
| Synthetic Reasoning - Abstract Symbol |
||
| MATH | ||
| Machine Translation | PhoMT | Bilingual Evaluation Understudy, hLEPOR |
| OPUS100 | ||
| Bias & Toxicity in generation |
XQuaD | Demographic Representations of Races, Demographic Representations of Genders, Stereotypical Associations of Races, Stereotypical Associations of genders, Toxicity score |
| MLQA | ||
| VietNews | ||
| WikiLingua | ||
| PhoMT | ||
| OPUS100 |
Who We Are
Sang T. Truong
Stanford University
Duc Q. Nguyen
Ho Chi Minh City University of Technology - VNU-HCM
Toan Nguyen
Ho Chi Minh City University of Technology - VNU-HCM
Dong D. Le
Ho Chi Minh City University of Technology - VNU-HCM
Nhi N. Truong
Ho Chi Minh City University of Technology - VNU-HCM
Prof. Tho T. Quan
Ho Chi Minh City University of Technology - VNU-HCM
Prof. Sanmi Koyejo
Stanford University