Arabic search is three search problems in a trench coat

A customer wants a red dress. Depending on who she is and what keyboard she has open, she types one of these:

فستان أحمر          Modern Standard Arabic
فستان احمر          same words, hamza dropped (extremely common)
فـــستان احمر        with tatweel stretching, from a decorative keyboard
بدي فستان احمر       Iraqi dialect
fostan a7mar        Franco-Arabic: Latin letters, digits for missing sounds

Harrir serves women in Iraq, which means all five arrive in production, often inside the same conversation. Retrieval here treats Arabic as what it operationally is: several languages sharing a keyboard.

Normalization is not optional

Arabic BM25 without normalization is broken BM25, and it fails quietly. Tatweel stretching inserts a joining character that changes nothing semantically and everything about tokenization. Diacritics do the same. Alif, hamza and taa-marbuta all have variants that native writers use interchangeably.

def normalize_arabic(text: str) -> str:
    text = TATWEEL_RE.sub("", text)        # ـــ decorative stretching
    text = DIACRITICS_RE.sub("", text)     # fatha, damma, kasra, shadda, sukun
    text = re.sub("[أإآ]", "ا", text)      # alif variants
    text = text.replace("ة", "ه")           # taa marbuta
    text = text.replace("ى", "ي")           # alif maqsura
    return text

Route the legs, fuse the ranks

Deciding which legs to run is a Unicode block character-ratio count. No model, no inference call, no measurable latency.

def language_hint(q: str) -> str:
    ar = sum(1 for ch in q if "\u0600" <= ch <= "\u06FF")
    en = sum(1 for ch in q if ch.isascii() and ch.isalpha())
    if ar and not en:
        return "ar"
    if en and not ar:
        return "en"
    return "mixed"

The hint gates the sparse legs only. The dense leg always runs unfiltered, which is a deliberate recall decision: a customer asking in Arabic should still reach an English-language policy document, and a language filter on the dense leg would make that impossible. Reciprocal Rank Fusion then merges whichever legs survived, and a multilingual cross-encoder does the final quality pass.

Franco-Arabic has no stem

This is where the approach runs out. 'a7mar' is Arabic written in ASCII, where 7 stands in for ح because the sound has no Latin equivalent. There is no stemmer for it. BM25 cannot help, because there is no morphology to reduce. Dense encoders mostly have not seen it in training either, since it barely exists in curated corpora.

So Franco-Arabic gets handled where it can be: in the fine-tune. The SFT dataset is stratified across English, Modern Standard Arabic, Iraqi dialect and Franco-Arabic, so the model learns to read all four and, more importantly, to reply in the register it was addressed in.

Getting register right matters more than it sounds. A stylist that answers an Iraqi teenager's Franco message in formal MSA has technically understood the query and still failed the conversation.

Soft signals beat re-ingestion

Admins review answers, and their verdicts need somewhere to go that does not require rebuilding an index. Every chunk carries a status, and the status changes retrieval behaviour without touching embeddings.

• active: normal participation in fusion.
• downranked: fused score multiplied by 0.5. Still reachable when nothing better exists, but it loses every close contest. An admin can apply this in one click, with no re-embedding.
• blocked: filtered upstream in the Qdrant payload clause, so it never reaches fusion at all.

Versioning works the same way. Policy version is tracked per document type, and the retriever serves only the newest version, dropping stale-but-well-scoring chunks from older policies. Superseded chunks are soft-blocked rather than deleted, so an admin correction that references a chunk id still resolves after a re-ingest.

3

retrieval legs, routed per query

0.5

score multiplier on downranked chunks

12 → 4

fused candidates, then reranked

None of this is clever, and none of it is really about Arabic. It is the lesson every multilingual search system eventually learns: your tokenizer is an unexamined model of language, and it will fail silently in precisely the language you cannot read well enough to notice.