King Fahd University of Petroleum & Minerals (KFUPM)
Current sign language recognition and translation benchmarks are commonly collected in controlled settings with fixed or studio-style cameras, which limits how well they reflect everyday mobile communication. This gap is especially visible for Arabic Sign Language (ArSL), the primary sign language used across Arab countries.
Isharah-Selfie is a large-scale Saudi Sign Language (SSL) dataset recorded entirely with front-facing smartphone cameras under unconstrained selfie conditions: one-handed signing, close-range viewpoints, partial body visibility, camera motion, and heterogeneous device resolutions. Every clip carries gloss annotations for CSLR and Arabic translations for SLT, and we define signer-independent and unseen-sentences protocols to test generalization. Benchmarks show that current RGB- and pose-based methods struggle in this setting, especially on novel sentence compositions.
Deaf signers commonly use the front camera to talk to deaf and hearing people and to share on social media. Members of the local Deaf community confirmed that one-handed signing is normal in short, informal smartphone interactions.
Illustrative sketch. Example: “thank you for help” signed two-handed (top row in paper, Fig. 2) vs. one-handed selfie (bottom row).
14 iPhones across multiple models, one Samsung, one TECNO; focal lengths 2.69–2.87 mm.
Only 2.7% of frames show both hands; 65.7% show a single hand.
Every clip has sign-level gloss sequences and natural Arabic sentence translations.
1,493 sentences designed with domain experts across seven everyday domains, refined and cross-validated by certified sign language experts.
Signers hold the phone in one hand and record 15–25 sentences per session. Each sentence gets ≥5 recordings; a certified ArSL expert records the reference.
Signers pause 2–3 s with hands out of frame; MediaPipe hand-detection gaps mark boundaries for semi-automatic sentence clipping.
Two experts annotate gloss and Arabic text. Clips are checked against the reference; deviations are re-recorded or re-verified.
8 Deaf, 6 hearing ArSL translators, 2 hard-of-hearing. 11 male, 5 female, aged 21–50. Some wear masks or niqab. All gave informed consent and were compensated.
Portrait orientation, practical selfie distance, signing hand and upper body kept visible, camera movement minimized. Varied clothing, backgrounds and lighting across sessions.
Re-recording when the upper body was insufficiently visible or hands left the frame, plus checks on sign order and semantic content.
Disjoint signers across splits: 11 train (5,997 videos), 3 dev (1,500), 4 test (1,500). Sentences overlap, so models must learn signer-invariant representations.
Dev and test sentences never appear in training (8,356 / 330 / 311 videos; 49 and 50 novel sentences). Known glosses must be recombined into new utterances.
| Gloss (CSLR) | SI Train | SI Dev | SI Test | US Train | US Dev | US Test |
|---|---|---|---|---|---|---|
| Videos | 5,997 | 1,500 | 1,500 | 8,356 | 330 | 311 |
| Unique sentences | 1,476 | 1,073 | 493 | 1,377 | 49 | 50 |
| Avg. length | 4.02 | 3.60 | 3.28 | 3.72 | 5.58 | 4.96 |
| Vocabulary | 961 | 814 | 497 | 959 | 113 | 129 |
| OOV (%) | – | 0.0 | 0.0 | – | 1.77 | 0.78 |
SI = signer-independent, US = unseen-sentences. Arabic-text OOV in the unseen split is much higher (41.35% dev, 30.37% test), reflecting greater lexical variability for SLT.
Nine published models, evaluated with their original configurations (mT0-xl replaces the LLM in SLT baselines to support Arabic).
| Model | SI Dev | SI Test | US Dev | US Test |
|---|---|---|---|---|
| RGB-based | ||||
| TLP | 55.87 | 55.30 | 74.51 | 75.92 |
| Swin-MSTP | 84.61 | 85.01 | 85.49 | 88.07 |
| USTM | 85.00 | 87.33 | 86.31 | 87.72 |
| CorrNet+ | 40.10 | 39.83 | 68.64 | 68.29 |
| SlowFastSign | 70.36 | 66.63 | 83.91 | 84.05 |
| Pose-based | ||||
| MSLR-Pose | 58.48 | 60.98 | 83.91 | 84.05 |
| VIPL-SLP | 69.10 | 67.09 | 80.69 | 79.44 |
Word error rate (%), lower is better. Best result in each column highlighted.
| Model · Split | Set | B1 | B2 | B3 | B4 | ROUGE-L |
|---|---|---|---|---|---|---|
| GFSLT-VLP | ||||||
| Signer-independent | Dev | 43.68 | 32.22 | 28.46 | 26.30 | 55.11 |
| Signer-independent | Test | 30.74 | 13.08 | 9.34 | 7.67 | 45.11 |
| Unseen-sentences | Dev | 23.44 | 1.96 | 0.60 | 0.31 | 38.99 |
| Unseen-sentences | Test | 23.01 | 5.35 | 1.17 | 0.51 | 20.70 |
| SpaMo | ||||||
| Signer-independent | Dev | 42.92 | 31.26 | 27.27 | 24.89 | 55.88 |
| Signer-independent | Test | 33.52 | 17.43 | 13.47 | 11.40 | 47.72 |
| Unseen-sentences | Dev | 22.35 | 1.32 | 0.45 | 0.24 | 37.97 |
| Unseen-sentences | Test | 21.15 | 2.93 | 0.77 | 0.37 | 37.77 |
Higher is better. BLEU-3/4 collapse on unseen sentences: n-gram coverage of dev/test in training is 64–76% for unigrams, 23–29% for bigrams, ~4% for trigrams, and 0% for 4-grams.
| Pretrain on | Test on | No finetune | Finetuned |
|---|---|---|---|
| PHOENIX-2014 | Isharah-Selfie | – | 91.52 |
| PHOENIX-2014T | Isharah-Selfie | – | 91.43 |
| CSL-Daily | Isharah-Selfie | – | 91.08 |
| Isharah | Isharah-Selfie | 99.67 | 85.11 |
| Isharah-Selfie | Isharah | 98.90 | 75.88 |
WER (%) with CorrNet+, finetuning only the 1D conv, BiLSTM and classifier layers. Even within the same sign language, two-handed and one-handed signing don’t transfer.
The BibTeX entry will be available once the paper is published.
Isharah-Selfie is available for non-commercial research under a signed usage agreement.
The authors would like to acknowledge King Fahd University of Petroleum & Minerals (KFUPM) for its support of this work. We also thank the signers and sign language experts who contributed to the dataset.