🔍 Mizan reranker
Arabic cross-encoder rerankers, built to reject near-identical traps · v3 · 306M · v3-Turbo · 150M
| 🔎 Query / الاستعلام | 📄 Candidates, one per line / المستندات، سطر لكل مستند |
|---|
Each case has four candidates, all full 3-5 sentence passages like a retrieved document: the passage that answers the question, a trap, a partially related passage and an irrelevant one. The trap is a near-copy of the answer that fails one detail the question asks about: a different entity, the opposite property, the effect instead of the cause, or a different number or year. Only the answering passage should come first.
Every case is also re-run with three different phrasings of the question, so you can see the result does not depend on how the question is worded. Both models pass all 11 cases here; the gap between the answer and the trap is reported in logits as well, because Turbo's calibrated scores sit higher on the sigmoid and so look closer together. Switch models above to compare.
Paste a long article and a question. The text is split into paragraphs and reranked, so the paragraph holding the answer should surface. The example below is 816 tokens over 12 paragraphs, and it contains a paragraph that states the opposite conclusion: the answer should still come first.
What this model is for
Mizan-Rerank-v3 is a 306M-parameter Arabic cross-encoder (the larger model of the current generation). It was trained to answer one question reliably: which of these passages actually answers the query? — when the wrong passages share most of the query's words.
Its hard negatives are adversarial traps: a negated ruling, a swapped entity, a shifted number or date, an exception applied to the wrong case. The Trap detector tab shows that directly.
The two models
| Mizan-Rerank-v3-Turbo | Mizan-Rerank-v3 | gte base | bge-reranker-v2-m3 | |
|---|---|---|---|---|
| Parameters | 150M | 306M | 306M | 568M |
| Held-out mean nDCG@10 | 0.875 | 0.826 | 0.752 | 0.777 |
| Hit@1, long-context adversarial | 0.735 | 0.570 | 0.314 | 0.336 |
| Contradicting passage ranked first ↓ | 16.9% | 32.5% | 63.8% | 56.8% |
| Throughput (RTX 4090, fp16, 512 tokens) | 2,487 pairs/s | 1,238 | 1,192 | 600 |
Held-out mean nDCG@10 is the mean over four held-out sets (MTEB NamaaMrTydi unseen subset, short adversarial, long-context adversarial, Multi-LLM). Charts below are from the selected model's card.
How to read the scores
The score is the sigmoid of the model's single output logit. Use it to rank passages and for coarse cut-offs. It is not a calibrated probability, so tune any threshold on your own data.
Honest limits
- The trap cases in this demo were stress-tested on both models after their passages were lengthened: the answer ranks #1 for all three phrasings of all 11 cases, in float32 and float16. On single-sentence candidates, which neither model was trained on, Turbo ranked a partial or irrelevant passage above the answer in 6 of 33 phrasings; with the longer passages it is 0 of 33.
- About the gaps. Turbo's scores are calibrated to a higher range, so on the sigmoid its candidates look closer together than v3's. Compare the logit gap, which is scale-free: across these cases the mean gap between the answer and the trap is +4.93 for Turbo and +4.70 for v3, but Turbo's smallest gap is +0.23 against v3's +1.11, so Turbo does have the tighter cases.
- On the full held-out adversarial sets the selected model still ranks a contradicting passage first in 33% of long-context and 20% of short queries. Check critical answers, such as religious rulings or medical and legal statements, rather than trusting the top passage alone.
- Trained on Arabic, on query + passage inputs up to 3,072 tokens. Longer inputs are truncated; the architecture accepts 8,192 positions but quality beyond 3,072 has not been measured.
- The adversarial test sets come from the same generation pipeline as the training data, so they measure the targeted skill rather than general-domain quality.