Crossing Linguistic Horizons

Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models

Fairness-Aware Text Classification Leaderboard

Models UiT-VSMEC PhoATIS
AC↑ F1↑ AR↑ ECE↓ A@10↑ AC↑ F1↑ AR↑ ECE↓ A@10↑
URA-LLaMa 70B 0.24 ± 0.02 0.14 ± 0.01 0.58 ± 0.01 0.26 ± 0.02 0.37 ± 0.06 0.15 ± 0.01 0.22 ± 0.03 0.31 ± 0.00 0.81 ± 0.01 0.13 ± 0.04
URA-LLaMa 13B 0.31 ± 0.02 0.11 ± 0.01 0.58 ± 0.01 0.23 ± 0.02 0.57 ± 0.06 0.01 ± 0.01 0.05 ± 0.02 0.58 ± 0.00 0.84 ± 0.01 0.00 ± 0.01
URA-LLaMa 7B 0.29 ± 0.02 0.11 ± 0.01 0.60 ± 0.01 0.12 ± 0.02 0.41 ± 0.06 0.00 ± 0.01 0.00 ± 0.00 0.55 ± 0.00 0.30 ± 0.01 0.01 ± 0.03
LLaMa-2 13B 0.18 ± 0.02 0.08 ± 0.01 0.55 ± 0.01 0.45 ± 0.01 0.44 ± 0.06 0.02 ± 0.01 0.01 ± 0.02 0.57 ± 0.01 0.90 ± 0.01 0.01 ± 0.01
LLaMa-2 7B 0.25 ± 0.02 0.11 ± 0.01 0.57 ± 0.01 0.22 ± 0.02 0.53 ± 0.06 0.02 ± 0.00 0.06 ± 0.01 0.57 ± 0.01 0.68 ± 0.01 0.01 ± 0.01
Vietcuna 7B 0.15 ± 0.01 0.05 ± 0.01 0.46 ± 0.01 0.85 ± 0.01 0.16 ± 0.04 0.04 ± 0.01 0.01 ± 0.00 0.77 ± 0.01 0.21 ± 0.01 0.07 ± 0.03
MixSUra 8x7B 0.40 ± - 0.36 ± - 0.72 ± - 0.53 ± - 0.79 ± - 0.81 ± - 0.58 ± - 0.96 ± - 0.14 ± - 0.91 ± -
Gemini Pro 0.48 ± - 0.38 ± - - 0.34 ± - 0.43 ± - 0.79 ± - 0.67 ± - - 0.73 ± - 0.68 ± -
GPT-3.5 0.44 ± 0.02 0.42 ± 0.02 - 0.30 ± 0.02 0.36 ± 0.06 0.68 ± 0.02 0.66 ± 0.03 - 0.62 ± 0.02 0.67 ± 0.05
GPT-4 0.49 ± 0.02 0.47 ± 0.02 - 0.35 ± 0.02 0.36 ± 0.06 0.83 ± 0.01 0.76 ± 0.03 - 0.77 ± 0.01 0.87 ± 0.04